DOCUKENT RESUME 

ED 342 348 HE 025 323 



AUTHOR 
TITLE 

INSTITUTION 

REPORT NO 
PUB DATE 
NOTE 

AVAILABLE FROM 



PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



A Feasibility Study. 
{ED), 



Korb, Roslyn 

Post secondary student Outcomes: 
Feasibility Report. 

National Center for Education Statistics 
Washington, DC. 

ISBN-0-16-036134--6; NCES-92-013 

Feb 92 

llBp. 

U.S. Government Printing Office, Superintendent of 
Documents, Mail Stop: SSDP, Washington, DC 
20402-9328. 

Statistical Data (110) 
MF01/PC05 Plus Postage. 

Certification? College Graduates; •College outcomes 
Assessment; Colleges; Comparative Analysis; Data 
Analysis; -Data Collection; Employment Statistics; 
Higher Education; Information Sources; ^Outcomes of 
Education; Postsecondary Education; Surveys; 
universities; Vocational Schools 



ABSTRACT 

This report presents the findings from a study 
designed: (1) to determine the current availability and utility of 
student outcome statistics in the nation; and (2) to examine the 
feasibility and desirability of nationwide institutional reporting of 
student outcome data* Following an introductory section on the 
purposes of the report and the approaches taxen, section II addresses 
the desirability of student outcome data, focusing on the rtility of 
the information and the necessary properties of outcome statistics. 
Section III explores sources of available data on postsecondary 
student outcomes and describes current capabilities for reporting 
institutional-level student outcomes, examining both institutional 
and state capabilities as well as other sources. Section IV discusses 
problems with institutional-level student outcome reporting, 
examining the quality of data that are currently available and the 
current and prospective uses of these data* Section v identifies 
those aspects of student outcomes statistics that require 
definitional specificity and describes data collection procedures and 
methodologies that could minimize the various problems of 
student-outcome reporting and produce useful and meaningful 
institution-level data. Finally, section VI summarizes the report and 
formulates conclusions concerning the desirability and feasibility of 
different approaches for obtaining student outcome information* 
Suggestions for specific actions that must be initiated if meaningful 
institution-level student outcome data are to be a reality for all 
postsecondary institutions in the nation are provided. Appendices 
include technical notes and summaries of definition working group 
meetings. (14 references) (GLR) 
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EXECUTIVE SUMMARY 

Tlie purpose of this study is twofold: to determine the current avuihibilitv nnd utility of student 
outcome statistics in the nation and to examine the fensibilitv nnd dcsinibilitv fi^r' nationwide 
institutional reporting of student outcome data. The findings will assist in respondinu to the 
iTiandate ot section l()3(c) of Title I of the Student-Right-to-Know and Campus Security (SRK) 
Act (PL 101-542) that requires the Secretary (of the U. S. Department of Educal'ion) in 
conjunction with representatives of institutions of higher education, to "analyze the feasibility 
and desirability of making available to students and potential students - 

(A) the completion or graduation rate of individuals at an institution broken down hv 
program or field of study; 

(B) the completion or graduation rate of an institution reported bv individu:.! schools 
or academic divisions within the institution; 

(C) the rate at which individuals who complete or graduate from the prc^gram of an 
institution pass applicable licensure or certification examinations required for 
employment in a particular vocation, trade, or professit>nal field: 

(D) the rate at which individuals who complete or graduate from an occupationallv 
specilic program and who enter the labor market following completion of o'r 
graduation from such a program obtain employment in the oc-cupation Ibr which 
they are trained; and 

(H) other institutional outcomes that may be appropriate." 

This study considers each of the specified student outcomes alonq with two other insiituli(.n:il 
outcomes - cognitive outcomes and earnings of graduates/completers. Additionally, rales of 
transfer from one institution to another are considered separately from maduation/Jompietion 
rates. " ' 

Methods 

Data and information from several sources are synthesized in this report. Thev include- (1) A 
survey of a nationally representative sample of po.stsecondarv institutions conducted by the 
National Cemer for Education Statistics (NCES). Its purpo.se was to determine institutional 
capability ior reporting student outcome statistics; (2) Results from a survey of hiuher education 
iiKStitutions concerning their student as.sessment activities. This .survey was conducted and 
reported by the American Council on Education (ACE); (3) Data from three studies on state-level 
capability for reporting student outcome statistics and on state uses of student outcome data; and 
(4) Results from previous attempts to obtain student outcome data from postsecondarv 
mst.tutions. In addition, numerous di.scussions were held with representatives from ail sector's 
01 the postsecondary education community to ascertain current thinkinu in the area of studem 
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Desirability 



When considered in conjunction with the literature on pc.stsecondary student outcomes, the data 
assembled for this study clearly indicate that having information on student outcomes from 
individual postsecondary institutions is desirable. A surprisingly large number of institutions and 
state postsecondary agencies have and can report student outcome statistics. For example, 79 
percent of postsecondary institutions indicated they determined and could report a graduation rate 
and 39 state higher education agencies indicated they could report or will soon be able to report 
graduation rates for in-state public institutions. 

The prevalence of these data suggests widespread interest in them and good faith efforts to 
collect and maintain them for instiiJtional use and/or state reporting. Additionally, there is ample 
evidence that student outcomes are of intere.st and could be useful to consumers and financiers 
of postsecondary education for both consumer protection purposes and institutional accountability. 

For .student outcome data to be useful for institutional self-evaluation or accountability purposes, 
however, they mu.st validly measure the phenomena they purport to describe and should produce 
unbiased and consistent results. For these data to be adequate and appropriate for consumer 
protection u.se, they .should also be comparable among reporting entities so that consumers have 
a common basis on which to make an informed choice. While it may be argued that some 
information is better than no information, from a consumer protection perspective, non- 
comparable or incomplete information may be worse than no information if it leads consumers 
to false choices. 

Current Availability and Utility of Student Outcome Data 

Judged on the basis of these criteria and the information and data gathered lor this report, it is 
evident that, regardless of their prevalence, no currently available student outcome data, including 
graduation/conipletion rates, are completely adequate for consumer protection purposes at a 
national level. Currently available student outcome .statistics either are not comparable among 
institution.s, institutional sectors, or states; or, they are incomplete, or unreliable. 

Lack of comparability was the major problem with currently available graduation/completion 
rates. Although the NCES survey of postsecondary in.stitutions found that 79 percent of 
postsecondiiry institutions in the nation determine and can report a graduation/completion rate, 
only 41 percent of all institutions use an optimum methodology (tracking a group or coht^rt ot 
entering students from initial enrollment at the institution to completion). Among public 2-year 
institution.s, only about 28 percent of institutions use this methodology. 

Even among schools that use a cohort methodology to determine a graduation/completion rate, 
the data suggest that the procedures for implementing this methodology vary substantially among 
postsecondary institutions. Specifically, although there seems to be a substantial degree of 
similarity in the definition of an initial cohort to track, the similarities evaporate in the process 
of tracking a cohort through to completion. Even the defmition of a graduate/completer varies 
somewhat among institutions. 
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Similarly, tests used by postsecondary institutions to assess students' college-level basic skills 
are not comparable. The ACE survey found that although 72 percent of higher education 
institutions (a subset of all postsecondary institutions) indicated they assess the basic skills of 
their students, only 18 percent believed their assessments results could be compared to those of 
other institutions. 

Problems with reliability, coverage and representativeness adversely affect the usefulness of 
currently available employment statistics and licensure/certification examination pass rales. Eighty 
percent of all institutions reported on NCES' institutional survey that they could determine the 
employment outcomes of their graduates/completers. Fifty-eight percent reported being able to 
determine if their graduates/completers obtained necessary and appropriate licenses or 
certifications. Most of these institutions also reported that they assess these post-completion 
outcomes by surveying their graduates some time after they have left the institution. Surveys of 
graduates tend to have low respon.se rates. I.ow response rates generally result in unreliable and 
inconsistent statistics. They may result in biased statistics as well if respondents and non- 
rcspondents have different demographic characteristics or if they are in different post-completion 
circumstances. 

Information on individuals who transfer from an institution is generally not available to the 
individual institution. When a.sked by the NCES survey if they could determine that a .student 
who left their institution actually transferred to another po.stsecondary institution, only about 12 
percent of all postsecondary institutions indicated that they ct)uld currently obtain this 
information. 

State po.stsecondary education agencies also collect and report institution-level student outcome 
data. A .survey of state higher education agencies indicated that 28 of these agencies could t>r 
will .soon be able to report graduation/completion rates for all in-.state, public 4-year institutions 
and 22 could or will soon be able to report them for the entire .state/system. A separate poll of 
state po.stsecondary vocati()nal oversight agencies indicated that 33 collected placement data on 
public postsecondary vocational program completers. Twenty-seven of these agencies, 
representing 46 percent of public 2- and less-than-2-year .schools, collected data on training- 
related placement and 12 reported collecting licensure information for about one-third of public 
post.secondary schools with vocatitinal programs. 

However, differences in definitions among institutional sectors even witliin a single stj'te were 
cited as being extremely problematic in interviews with representatives of state vocational 
oversight agencies. This is apparently due to the segmentation of oversight responsibilities within 
states. For example, in only eight states are public 2-year, public less-than-2-year and private, 
for-profit schools the responsibility of the same agency. Thus, while state higher education and 
postsecondary vocational education oversight agencies have a .substantial amount of data that are 
related to student outcomes, the comparability of these data among and even within states is 
questionable. 
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Feasibility of Collecting Valid and Comparable Outcome Data 

For most student outcomes that were examined in this report it is possible to iucntil'y bnsic 
definitions and methodologies for developing comparable, consistent and valid institution-level 
student outcome statistics based on currently used best practice. For two of them, however, 
identifying a reasonable approach for achieving comparable and complete data was not possible. 

Graduation/Completion Rates 

The best practice for determining institution graduation/completion rates can be implemented by 
individual institutions. It requires each institution to: adopt the same cohort mcthodoltigy lor 
determining the rate; employ a standard definition of which students to include in the cohort and 
in calculating the rate; and report the rate by the same student and institutional characteristics. 

For example, .separate cohorts could be established for degree-.sceking full-time, part-linu\ anil 
transfer .students upon initial entrance to the institution. In.stitutions could report graduation rates 
of these cohorts by meaningful, relatively static student characteristics such as gender, 
race/ethnicity, age or intended major field of study at the time of initial enrollment. On the other 
hand, institutions would find it difficult, if not impossible, to report graduation/completion rales 
by actual major field of study, academic division or any variable that changes frequently over 
a .short period of time. 

That postsecondary institutii>ns throughout the nation can readily adopt data coUectinn and 
reporting .standards, once they are established and disseminated, has been dcmonstraied thnnigh 
the collection and reporting of other intra-institution statistics. 

Implementing the best practice for tran.sfer rates and employment outcomes, on the other hand, 
can be achieved most effectively and efficiently at the state level through state student unit-record 
.sy.stems that encompa.ss all institutional sectors (both public and private). 

Transfer Rates 

Determining transfer rates t)f students from one institution to another, as distinct from 
graduation/completion rates, is possible, but is highly dependent on at least .siiite-level 
involvement. Statewide student record systems that can track students as they transfer from any 
in.slitution included in the statewide .system to any other institution in the system have been 
operational in .several states for a number of years although the institutional coverage varies 
among states. Alternatively, electronic iran.scripts through which machine-readable performance 
and enrollment information about individual students can be transmitted from one institution to 
another is a developing method for tracking transfers. However, the efficacy of this approach 
has not been demonstrated. 
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Employment Outcomes 



For determining valid, consistent and comparable employment outcomes (employment riites and 
earnings) of graduates/completers, an efficient and cost-effective melhodoJogy is through 
electronically linking state files containing graduate/completer school records with available 
Unemployment Insurance (UI) Wage-Record Data. This methodology is currently being used by 
one state to determine employment outcomes of postsecondary completers in public sector 
institutions and has been tested in 13 other states to determine the employment outcomes of Job 
Training Partnership Act program completers. It is estimated that this method a.^u: t can account 
for between 70 and 90 percent of all postsecondary education completers in a state. For more 
comprehensive coverage, the UI data would have to be supplemented with information from 
employers not included in the UI system, with UI data from neighboring slates, and, to determine 
the relatedness of completers' employment to their education, with surveys of employers 
identified through the UI records. 

Comparable and complete data for licensure/certification pass-rates and postsecondary 
as.sessments would be extremely difficult to achieve under any circumstances. Moreover, there 
is currently no best practice on which to model such complex data collection. 

Licensure/Certificati{>n Rxaminalion Pass Rates 

Institution-level pass rates on licensure or certification examinations would be consistent and 
somewhat comparable indicators of student achievement, especially when these tests are 
administered at a national rather than a stale level. However, the uneven coverage of 
licensure/certification retjuirements across fields of study and the difficulties in obtaining 
institution-level pass rates from the multitude of licensing and certification bt^ards are currently 
prohibitive obstacles, particularly for schools with multiple programs. Additionally, many 
licensure/certification examinations are administered at the state level with each slate having its 
own examination and criteria for passing. Thus, even if the data were collectable, their 
comparability would be problematic. 

Assessment Outcomes 

For assessment of student learning or cognitive skills, it might be possible to develop some type 
of postsecondary assessment (e.g., a test of posfsecondary-level basic skills or tests of subject- 
matter achievement) that possibly could be consistently applied to all postsecondary institutions 
regardless of the entering level skills of their students. However, the relevancy of basic skill 
testing to all postsecondary institutions in the nation would be doubtful; and, clearly different 
subject-matter achievement tests would have to be developed and applied to different sectors of 
postsecondary institutions. Moreover, either of these assessment approaches would be a long- 
term, costly effort, requiring a substantial degree of research, development, and consensus- 
building. 
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Additional Ccnslderations 



Two additional suggestions are warranted if graduation/completion rates and other institution- 
level student outcomes are to be as meaningful as possible to all potential users. 

Need for Contextual Information 

Student outcomes are related not only to the efforts and effectiveness of the postsecondary 
institution, but also to the characteristics of the students in the institution. Thus, it is vital that 
contextual information describing the institution's student population be considered in conjunction 
with its student outcome data. Indeed, t may be that for truly valid comparisons among 
po.stsecondary institutions, institutional outcome .statistics should be adjusted to account for 
differences in characteristics among the students served by the institutitin. 

Need for Facilitating the Collection of Comparable Data 

The different requirements for data collection and reporting indicated in sections 103 and 104 of 
Title I of the SRK Act distinguish postsecondary institutions on the basis of their athletically- 
related student aid policies.' This distinction will tend to perpetuate the non-comparability of 
student outcome statistics among postsecondary institutions and should be reconsidered. If 
information, such as graduation rates by gender or ruce/ethnicity, or average 
graduation/completion rates are useful, they should be available to all potential students and their 
families and from all postsecondary institutions. 

Conclusions 

Clearly, implementing the suggested best practice will require the investment of resources 
(personnel, funds, and sufficient time). In some cases it will also require an enlarged state role 
in data collection and reporting. As a result, there Is little hope that even graduation rates as 
identified in the SRK Act can be fully established for reporting in 199.^, as specified in the 
Student-Right-to-Know Act. The most realistic expectation is that institutions can immediately 
begin tracking entering students and disclose or report persistence rates until the first cohort 
matures. (For 4-year in.stitutions a cohort of entering students would not mature until 1998 or 
150 percent of the normal 4-year program length.) 

For the other specified outcome statistics (i.e., employment outcomes, licensure pass rates) as 
well as transfer rates and cognitive outcomes, the process of establishing them nationwide will 
be more difficult and will take more time. Each of them requires some developmental effort and 
involves a considerable degree of cooperation between postsecondary institutions, state 
postsecondary agencies, and federal agencies. However, if this investment were made and 
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sufficient cooperation could be established, reporting of nationwide institution-level student 
outcome data that would be useful for consumer protection and other comparative purposes is 
possible and would be worthwhile. 




FOREWORD 



The impetus for this report was the Student-Right-to-Know and Campus Security Act (PX. 101- 
542) enacted in November, 1990. Establishing guidelines to implement the Act is the 
responsibility of the Office of Postsecondary Education in the U.S. Department of Education 
(ED). Because this legislation has significant implications for postsecondary education data 
collection and reporting, however, the National Center for Education Statistics (NCES) has 
participated in the Department's implementation efforts. This role was supported by NCES' 
traditional role to provide technical leadership and guidance in defining nationally uniform 
measures for the collection and reporting of postsecondary education data. 

One aspect of NCES' role was to facilitate discussions of the provisions of this legislation among 
the Department and representatives of a wide variety of postsecondary institutions and 
organizations. In so doing, NCES utilized the mutually beneficial links it has established through 
its data collection activities with the postsecondary education community and especially state 
education agencies and postsecondary institutions. 

A second aspect of NCES* role was to provide a foundation for the Department of Edi^' /ion to 
consider the desirability and feasibility of postsecondary institutions' reporting various student 
outcomes. Such an analysis is mandated in Section 103 of Title I of the Student Right-to-Know 
AcL NCES collected data and synthesized information and data from several other sources to 
examine these dual questions. This report represents the results of that work. 

We hope that this report will provide the postsecondary education community with sufficient 
information and thought-provoking conclusions to enable discussions of student outcomes in 
postsecondary education to move forward on a coherent and productive path. We look forward 
to that discourse. 



Emerson J. Elliott 

Acting Commissioner 

National Center for Education Statistics 



Paul R. Hall 

Acting Associate Commissioner 
Postsecondary Education Statistics 
Division 
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SECTION I 



PURPOSE OF THE STUDY AND APPROACHES TAKEN 

A. Objectives 

The purpose of this study was twofold: to determine the current availability and utility of 
student outcome statistics in the nation and to examine the feasibility and desirability for 
institutional reporting of student outcome data. In st> doing, it will assist in responding to the 
mandate of section 103(c) of Title I of the Studcnt-Right-to-Kjiow and Oimpus Security (SRK) 
Act (PL 101-542) that requires the Secretary, in conjunction with representatives of institutions 
of higher education, to "analyze the feasibility and desirability of making available to students 
and potential students - 

(A) the completion or graduation rate of individuals at an institution broken down by 
program or field of study; 

(B) the completion or graduation rate of an institution reported by individual schools or 
academic divisions within the institution; 

(C) the rate at which individuals who complete or graduate from the program of an 
institution pass applicable licensure or certification examinations required for 
employment in a particular vocation, trade, or professional ileld; 

(D) the rate at which individuals who complete or graduate from an occupationally specific 
program and who enter the labor market following completion of or graduation from 
such a program obtain employment in the occupation for which they are trained; and 

(E) other instiluti(mal outcomes that may be appropriate." 

B. Definitions and Treatment of Feasibility and Desirability 

Desirability addresses the importance of the data and how and by whom they will be used. Feasibility 
concerns the reasonableness and practicality of gathering and reporting data. 

While the SRK Act implies a distinction between feasibility and desirability, these two concepts are 
inexorably intertwined. Desirability is concerned with data utility in relation to specific uses and users 
as well as the data's generic importance. An analysis of the feasibility of collecting and reporting data 
must be carried out in the context of these specific u.sers and uses. Furthermore, collecting and 
reporting data require resources. If data are not important, expending limited resources to collect and 
report them is not sensible. On the other hand, it may not be possible to collect and report 
demonstrably important data appropriately and adequately to satisfy the need for information. 
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The SRK Act, for example, was intended as consumer protection legislation and makes the assumption 
that potential students and their families would use the information on student outcomes for comparing 
institutions and choosing among them. That students would actually use the information for this 
purpose is debatable. The feasibility question, however, is whether or not data can be provided that 
would be adequate for this purpose and appropriate for this specific use. 

The very existence of Title I of P.L. 101-542 suggests the desirability of institution-level student 
outcome data. That Congress passed legislation requiring all postsecondary institutions* to disclose 
graduation rates for their students speaks to the public policy importance of this statistic. The 
additional reporting requirements in section 104 of the SRK Act for institutions that award athletically 
related student financial aid and the outcomes specified for the feasibility study in section 103(c) are 
indicative of a strong national interest in additional student outcomes. Furthermore, as this report will 
demonstrate, there has been a call for postsecondary student outcome information for a substantial 
period of time and from a broad spectrum of the postsecondary education community. 

C. Assumptions 

Tlie statement in the SRK Act that "knowledge of graduation rates would help prospective students and 
prospective student athletes make an informed judgment about the educational benefits available at a 
given institution of higher education" indicates that student outcome data should be feasible nationwide 
at the individual in.stitution level, rather than at just a stale or national level. 

D. Approaches Used in this Report 

This report focuses finst on specific uses that could be made of institution-level student outcome data 
and current capabilities and methods of reporting institution-level student outcomes. It then evaluates 
the adequacy and appropriateness of currently available outcome data in relation to specific uses and 
describes some of the problems that have historically been encountered in reporting these data. Finally, 
this report describes how these specific student outcome statistics could be collected and reported to 
meet specific user needs given current technology. 

In the process of addressing the specific outcome measures indicated in the SRK Act (i.e., training- 
related employment, licensure and certification pass rates, and other student outcomes), .several 
additional concerns are broached in this report. These include: (1) the feasibility, desirability, and 
problems with current reporting requirements of the SRK Act, particularly graduation/completion rates, 
the exclusion of degree-seeking part-time students, and the inclusion of transfers in the definition of 
"completer"; (2) the need for consistency between .sections 103 and 104; and (3) the need to provide 
contextual data in reporting student outcomes. 



^1» SKK Art speeir*s eligible instiiuiiww of higher ettucition. This has been deHned in this report as all pcstswcondiry institmiwis In the anion th,ii .w 
elijsihie fw sodcni fitwicial aid {Pell grants, Sl.>fford Iviiuis, or cimpuR-b.ised aid) under Tttic IV of the Higher Education Act of l«Hi5. »s amended. 
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Organization of the Report 



Section II addresses the desinibility of student outcome data. It highlights the generic uses ot 
these data and the evolution of these uses as expressed in both the literature and national 
legislation. It describes specific uses and users of this type of data from an historical as well 
as a current perspective, and it provides some insights on prospective uses particularly how 
institutions themselves could make use of this information. It also suggests the technical 
characteristics a statistic must have to make it adequate and appropriate for a specific use. 

Section III explores sources of available data on postsecondary student outcomes and describes 
current capabilities for reporting student outcomes at an institutional level - both institutional 
capabilities and state capabilities. 

Section IV examines the fit of data that are currently available and current and prospective uses 
of these data. It describes problems that have been encountered in previous attempts to 
establish nationwide student outcome data. It also describes problems with currently available 
data. 

Section V suggests those definitions and data collection procedures and methodologies that 
could minimize the various problems of student-outcome reporting and produce useful and 
meaningful institution-level student outcome data. 

Section VI summarizes the report and formulates conclusions concerning desirability and the 
feasibility of different approaches for obtaining .student outcome informatitin. It also suggests 
activities that must be initiated if meaningful institution-level student outcome data are to be 
a reality for all post.secondary institutions in the nation. 
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SECTION II 
DESIRABILITY OF STUDENT OUTCOMES 



A. Utility of Student Outcome Information 

A review of the literature and previous national postsecondary and vocational education legislation 
sugtjests that uses of student outcome information have been considered from two different primary 
perspectives: institutional accountability and consumer protection. These two perspectives are related, 
but the approach to student outcomes that each would dictate is somewhat different. 

Institutional accountability focuses on the mission of the institution and the goals the institution hopes 
to meet. The interest and call for increased accountability for postsecondary institutions evolved in 
conjunction with the more pervasive movement of management by objectives.'' As a consequence, 
accountability, for the most part, measures institutional performance (and student performance) against 
the institution's mission and goals and, possibly, the state's goals for that institution and postsecondary 
education in the state. 

Consumer protection, at least in the narrow sense of the student as the consumer, takes a much more 
student-oriented approach; it is concerned with the outcomes the student expects as a result of 
participating in postsecondary education. It virtually disregards the mission of the institution except 
in those in.stances where the explicit goals of the institution coincide with .student expectations. 

In 4-year degree granting institution-^, for example, the primary mission has traditionally been viewed 
as providing instruction in academic courses and programs. Student persistence in, and eventual 
completion of, the.se programs has been considered as much the responsibility of the student as the 
responsibility of the institution. From this institutional perspective, institutional performance would 
be measured in terms of the effectiveness of the instruction in imparting knowledge and cognitive 
skills. While gains in knowledge and cognitive skills resulting from the interaction of in.stitution and 
.student effort are relevant to .students and prospective students, consumer protection advocates might 
argue that they may not be quite as important as the expectation that completion of a higher education 
program will result in an economic advantage for the student in terms of occupation and level of 
earnings. Thus, from an accountability perspective, the most desirable student outcomes may be in the 
realm of learning and the development of cognitive skills, while from a consumer protection 
perspective they would be the experiences of the individual in making the transition from school to 
work. The SRK Act takes the perspective of consumer protection. It is predicated on the belief that 
students need certain information to make an informed choice among po.stsecondary institutions. 
However, whether students would actually use institution-level student outcomes information in 
choosing a po.stsecondary in.stitution has been questioned. 



Asttn. A. MMSurinfs ihe Ouicnmes of HtRhcr Filufalipn in Nfw DiiyciUms for lnslitulion.il Research. IIuamtiI R. RowtfL Rd.. C.iliffimi.i: 
J<iss«y-li.'ihs, V.I no. 1, Spring I*(74, pp, 23-46, 
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1. Student uses of student outcome information 



To gain some insight into what influences students in selecting a postsecondary institution, data from 
two national surveys of students enrolled in postsecondary institutions were examined.'' In the 1986-87 
and 1989-90 academic years, NCES asked students why they decided to enter the school they were 
attending. Students could rate several possible factors that could have influenced their enrollment 
decision. 

Table II- 1 presents the percent of students that reported a particular reason was very important by the 
type of institution students were attending. As it shows, 75 percent of the students in 1987 and 67 
percent of the students in 1990 responded that the school's a>urse of study was very important in their 
choice of school. This was true of students enrolled in all po.stsecondary institutional sectors from 4- 
year, degree-granting institutions to less-than-2-year private, for-profit technical/vocational schtn)ls. 
Forty-eight percent in 1987 and 46 percent in 1990 responded that the school's "good reputnlion" w;is 
important, although reputation was important to a higher percentage of students enrolled in privnte 
sector schools (both non-profit and for-profit) than in public sectors schools. Thus, significant 
proportions of students believed that getting a good education in their field of interest was important. 

Thirty-six percent of students in 1987 and 33 percent of students in 1990 reported that the school's 
reputation for placing its graduates was very or .wmewhat important in their choice of school. 
However, over half the students in private, non-profit schools in both 1987 and 1990 felt this was an 
important consideration, and over 63 percent of students in private, for-profit schools in 1^)87 and 5(> 
percent of students in 1990 reported this was an important reason. 

More immediate, primarily economic reasons for choosing a postsecondary institution in addition to 
its program selection, its reputation, and its ability to place graduates were also considered important 
by significant numbers of postsecondary education students in both 1987 and 1990 (table ll-l). 

While students are the most obvious consumers of postsecondary education, they are certainly not the 
only consumers. Any individual or organization that has a financial interest or other stake in the 
student's persistence, progression, and completion, such as the student's parents, spouse, or empUiyer 
could also be considered a "consumer." In fact, in this broader sense, the consumer would include the 
federal government, as well as state and local governments, since they provide financial assistance to 
students directly through student financial aid and indirectly through funding allocations to 
postsecondary institutions. 

2. State and other uses of student outcome data 

Many states, in addition to being "consumers" of postsecondary education, have oversight 
responsibilities for some or all sectors of postsecondary education institutions In their state, so their 
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concern over student outcomes has a strong accountability component as well as a consumer protection 
interest. As a result, the uses that state agencies make of student outcome information provide a 
somewhat different perspective on the utility of these statistics. 

In a recent survey of state postsecondary vocational oversight agencie«;, responses of individuals in 
these agencies to questions concerning uses of student performance data fell into three categories:** 
consumer rights, licensure, and management. Fourteen states make community college student 
performance data available to the public; 17 states use student outcomes in decisions to renew licenses 
to operate for-profit institutions; 22 states use this type of information in funding decisions; and several 
states use it for program review to assess needs for improving operations (1.3 states), expanding or 
contracting specific occupational programs (21 states), or taking corrective actions (18 states).^ 

Clearly, states are significant users of student outcome data. Their reported uses cut across both 
consumer protection and institutional accountability. States also reported using institution-level student 
outcome data in assessing needs for improving institutional operations. This use is particularly relevant 
to institutions. It permits states to work in partnership with institutions in making sound management 
decisions about program offerings, enrollment size, and program quality. 

Notably, Ewell and Jones^ argue that one of the most important uses of institution-level student 
outcome data is that made by institutions themselves. Such data have a multiplicity of uses for 
institutions' .self-assessment and evaluation activities. 

In addition, it is of interest to examine the utility of student outcome information in the context of 
current federal initiatives in postsecondary education. These would include the national education goals 
that emphasize economic development and global competitiveness** and an increasing emphasis on 
institutional accountability at the national level as demon.straled by the Department of Education's 
.student loan default initiatives. Additional u.sers of institution-level student outcome statistics may be 
identified. High school counselors could use these data to help students select a postsecondary 
institution. Taxpayers might use them to determine the return on their investment in publicly supported 
po.stsecondary education; or potential employers of post.secondary school completers might use 
information on the ab.solute level of, or relative gains in, knowledge and skills made by students in 
specific programs among the different in.stitutions to target recruiting efforts more effectively. 

Furthermore, reliable national data on degree production, time to degree, and, nnire importantly, the 
nature and quality of student learning, could facilitate the development of policies to meet regional and 
national manpower needs for continued economic growth. 
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If one accepts the suggestion that institution-level student outcome information is used, or can 
potentially be used by all postsecondary education constituents - from participants to providers to 
financiers (federal, state, and local government), the obvious next step is to connect specific student 
outcome statistics to each of the many and varied users and uses. It is important to identify the 
properties each outcome statistic must have to assure its adequacy for a particular use and its 
meaningfulness to potential users. Unfortunately, this discussion must rely heavily on a conceptual 
treatment since there is little empirical evidence to support it. 

Precisely which student outcome statistics should be considered in this exercise could be endlessly 
debated. However, the most obvious approach, and the one taken here, is to use the outcomes 
identified in the SRK Act (sections 103(a), 103(c) and 104 of Title I) since they are fairly diverse and 
encompass a wide range of outcomes.' Additionally, the "other institution outcomes that may be 
appropriate" provision permits two additional outcomes to be considered, one of which is more 
important in a consumer protection context, i.e., earnings of graduates/completers, and the second, 
which is more important from an accountability perspective, i.e., results of postsecondary assessments. 

B. Necessary Properties of Student Outcome Statistics 

For any statistic to have meaning, it must be a valid measure of the phenomenon it purports to describe 
and it must produce unbiased and consistent results. While these three criteria seem simple enough, 
oftentimes they are difficult and/or very costly to achieve. Indeed, many of the problems that have 
been identified in previous attempts to require nationwide, sub-national, student outcome reporting have 
resulted from shortcomings in one or more of these properties. However, even though these criteria 
are absolutely necessary for any statistic, they are not sufficient for all the diverse uses an outcome 
stati.stic might have. 

In the context of institutional accountability, if the uses were limited to reviews of institutional 
operations and programs, then stntistics that produce valid, consistent, and unbiased results would 
probably be sufficient. Further, for most state oversight agency uses (i.e., licensure decisions, funding 
decision.s, decisions to expand or contract specific occupational programs, or identification of the need 
to take corrective action), having consistent, unbiased student outcome statistics that are relevant for 
a particular institution js, in most cases, tenable. 

However, if institutions base at least a part of their self-evaluation on comparisons with peer 
in.stitutions or if states make any of their program and funding decisions by comparing among 
institutions with similar missions, then comparability of outcome .statistics targeted toward the similar 
missions of these "peer" institutions is necessary as well. If some states make these decisions by 
comparing all institutions in the state, then comparability of outcome statistics within that .state is 
necessary, regardless of the institutional mission. 
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From the consumer protection perspective, the comparability of institution-level student outcome 
statistics is necessary if students and prospective students are to have a valid foundation for choosing 
among postsecondary institutions. While this statement may seem axiomatic, achieving agreement on 
it among the various postsecondary institutional advocacy groups was difficult since it involved 
concurrence on the need for procedural and methodological comparability as well as definitional 
comparability. There was strong doubt that this level of comparability should even be attempted, given 
the diversity of missions, and characteristics among the postsecondary institutions in the nation. "tHIs 
doubt persisted even though the need for comparable information was finally recognized."' 

Comparability of student outcome statistics among postsecondary institutions is also important for 
federal initiatives. In using them to assess progress toward the national postsecondary education goals, 
student outcome statistics must not only be comparable among postsecondary institutions but they must 
be comparable over time as well. For employers to use student outcome statistics to target recruiting 
efforts, or for states to make specific programmatic decisions, comparability of outcomes amonu 
specific piograms among institutions becomes crucial. 

Completeness of coverage among institutions and institutional sectors'^ in reporting comparable student 
outcome statistics must also be considered in determining the utility of these data.'- Clearly, for 
institutional use of siudent outcome data, coverage is not at issue. Institutions have complete discretion 
in doing comparative analyses and in choosing their peers for comparative purposes. For slate use, the 
coverage of institutions depends upon how the states use the information. Needs for coverage may 
vary from a single institution to all institutions in a single sector (e.g, 2-year community colleqes) to 
all higher education institutions (2- and 4-year accredited institutions), to all institutions oneralinu in 
the state. ' * 

For federal use such as the student loan default initiative, comparable outcome statistics are required 
from all institutions that purport to have the same mission, regardless of institutional sector. A 
nationally representative sample of postsecondary institutions by sector would suffice for nssessinu such 
national goals for postsecondary education as increased graduation/completion rates or 1)cttcr 
performance on tests of cognitive skills. 

While the technical quality of student outcome statistics must be established, it is also important that 
the mformation provided by the data be as meaningful as possible. In judging the efficacy of an 
mstitution, both consumers and funding agencies should be aware that student outcomes are related not 
only to the efforts and effectiveness of the postsecondary institution, but also to the characteristics of 
the students m the institution. In fact, much of the research on identifying correlates of student 
outcomes has shown that student characteristics are a better predictor of student outcomes than 
mstilutional characteristics or processes. Thus, it is vital that contextual information describing the 
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institution's student population be considered in conjunction with its student outcome data. Indeed, 
it may be that for truly valid comparisons among postsecondary institutions, institutional outcome 
statistics should be adjusted to account for differences in characteristics among the students served by 
the institution. 

C. Summary and Conclusions 

Two primary uses of institution-level student outcome statistics were identitled-institutional 
accountability and consumer protection. Other uses were identified as well. For many institutional 
accountability purposes, student outcome statistics that are relevant to the institution and that produce 
valid, consistent, and unbiased results are sufficient. However, to evaluate institutional performance 
against that of other institutions, comparable student performance statistics are needed at least among 
institutions that are being compared. To provide students, prospective students, and their parents 
sufficient information to make an informed decision concerning the choice of a postsecondary 
institution, a comparable set of student outcome indicators from all postsecondary institutions in the 
nn'iion is necessary. 
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SECTION III 



CURRENT STATUS OF INSTITUTION-LEVEL 
STUDENT OUTCOME REPORTING 



A. Introduction 



Intormation about the current status of student outcome data at the institutional level is important 
because it may answer the question of what data might be feasible for institutions to report at the 
present time. Clearly, if institutions already have certain student outcome data, then disclosing or 
reporting these data would be possible. Institutions, however, are not the only repository of institution- 
level student outcome data. State higher education agencies oftentimes maintain individual student 
records that have been obtained from institutions. Some of these agencies are able to determine 
relationships between the student and his/her outcome and provide feedback to institutions. Other state 
oversight agencies, institution and program accrediting bodies, and licensure and certification boards 
also may have data on student outcomes by institution, and in some cases may inform institutions of 
their students' performance. Tlierefore, information from several diverse sources was obtained to 
develop a more comprehensive picture of the current status of collecting and reporting student outcome 
data. 

B. Sources of Data on Current Status 
I. Institutions 

An obvious source of information on the existence of institution-level studen* outcomes is the 
po.stsecondary in.sthution itself. To tap this source of information, NCES conducted a Fast Response 
Survey ot a nationally representative sample of postsecondary institutions. All postsecondarv 
institutional sectors, from 4.year universities to for-profit trade and technical schools were represented 
m the sample of over m) institutions. The questionnaire included items on three topics: what student 
outcome data institutions have and maintain, if any; what procedures and methods are used to gather 
these data and/or report the information; and to which students do these outcome data apply 
(Technical information on this survey is available in appendix B.) 

The NCES Fa.st Response Survey was framed around the student outcomes specified explicitly in 
section 103 (i.e., graduation rates, graduation rates by academic discipline or department within' the 
in.stitutions, training-related employment, and licensure p;iss rates). Since institutional assessment of 
studeiits is not specifically mentioned for study in section 103, institutional assessment was not 
included as an element in the Fast Response Survey. However, the "other outcomes" provision in 
section 103, coupled with the current interest in postsecondary student assessment, particularly for 
institutional accountability purposes, warrants consideration in this report. 

To obtain estimates of the number of higher education institutions that conduct student assessments 
and to obtain some sense of the content of these assessments, results from the American Council on 
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F.lucntion's (ACE) 1990-91 survey of higher education institutions for its Campus Trends report were 
^d s su^^^^ asked a nationally representative sample of public and private 2- and 4-year h.g er 
^ducatio^Sns^^^^^^^^^ they Assessment activities underway or planned, the content of the 
assessment, and some uses they make of the assessment data. 



2. States 



As mentioned above, many stales maintain managemem infonnation systems (MIS) on students enrolled 
fn t" S po^tsecondary institution. Information on individual students is ly ob.a.ned from 

e-,ch institution covered by the state MIS and is then mcorporated into a central data base, ho 
example a «me agency might receive a list of all students enrolled for the first-time tn an academ c 
veT ^ ons wid^ characteristics of the students and an identification number for each student. The stat 
S S hen establish a cohort" comprised of these students, track the students- subsequem 
en ollmem spells for some specified period of lime, and eventually determine a graduafot, rate f™ he 
Denning and using a cohort of students permits slates to determme wh.ch -d h""y 
students who enrolled in a given year graduated. Thus, states may also be able to <i<=t"™>ne a 
S'^ol rite for an institufion. They may be able lo determine if a * " -™ f„ X.^: 

institution in the state sradualed from any other in.«ilulion .n the stale system O her student outcome 
la in addition to gra-duations or transfers, might be similarly handled by the .state agency. 

The nrecise data that are collected for a state system, the methodology and procedures ussed the uses 
J^l'^of L data the institutional coverage in the state, and the stale's rcporfng capabdifes vary 
TonlweLMy among the slates. As a result, it was vital for .his report to obtain as much tniormat.on 
as possible about .state capability for reporting institution level student outcomes. 

Thus a second source of national data on the current status of institution-level student outcome data 
was from s™.e higher education agencies. In January 1991, a survey of 50 state h.gher educ^tum 
es w Tconducted by the nttional office of the State Higher Education Executive Ott.cers 
SHFEOr The purpose of the .survey was to determine the data collection and reporting capab.ht.es 
n pface at the stTlevel that would relate to institution-level student outcome data. Questions were 
sked concerning the specific studem outcome data that were collected and reported by the s a e 
incy! if any; the methods and procedures used in collecting and reporting these data; how the data 
were used; and when these kinds of data were first collected. 
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"li,.Kh,»» li T,.„d». mi. HW„, f N-.*" 81. Am^ric.™ C,™!! „ w,«hinj...„. I>.C.: 

"a cohort is ^nncd as a gr^up of individuate (s.ud..nts) Uvu have b comnroo time W chamcterislic. such a. age. a«mbe« of a gr«da,.ing cb.-^ ^ 
members of an entering class of sludmts. 

'•Umh. CS. and Rus«n. A. B. "Statewide Student Data Systems emJ CapabiliUes to Report P«tteoondaiy Oraduation Rate*'. SHEK5. 1991. 
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Results from another study of state agencies that had been conducted in conjunction with the reporting 
requirements of the Vocational Education Act of 1989 were also used as a source for this report. In 
1990, the Office of Planning. Budget, and Evaluation (OPBE) in the Department of Education 
supported a study to ascertain the reporting capabilities of state agencies responsible for vocational 
education in each of the 50 states. The data were collected and a report was prepared by Research & 
Evaluation Associates, Inc. (REA).*^ The study attempted to answer three questions: 

1. Do state vocational education agencies collect information on student outcomes? If so, 
what types? 

2. How is information on student outcomes obtained? 

3. How is information on student outcomes used? 

REA relied on two types of information for their study: (1) state reports and legislation pertaining lo 
the measurement of student outcomes for postsecondary institutions that provide vocational education; 
and (2) an average of five telephone interviews per stale of officials in the agencies responsible for 
vocational education in the state. The study was limited to three categories of postsecondary 
institutions: public community colleges, public vocational/technical schools, and proprietary (for-profit) 
vocational/technical schools. Public community colleges and public vocational/technical schools were 
considered a single entity in states where the oversight agencies were the same. 

To obtain additional detail on the specific procedures followed in reporting vocational student outcome 
data, NCES conducted telephone interviews of vocational oversight organizations in nine randomly 
selected states (Arkansas, Colorado, Connecticut, Delaware, Mississippi, Nebraska, New Jersey, Rhode 
Island, and Virginia). One purpose of the interview was to collect more information on how they 
calculate placement rates (if they, indeed, obtain and report these rates) and how they determined 
whether student placements were related to their training. Another purpose was to get a sense of the 
effect that state privacy laws might have on reporting licensure pass rates back to an in.stitution or at 
an institutional level. States that did not report either placement or licensure pass rates were asked to 
explain the barriers they perceived in reporting these data. 

3. Other Sources 

While it was not possible to identify every organization that might have data on institution-level student 
outcomes, both state licensure boards and accrediting agencies were thought to be repositories of such 
information. For licensure boards, this was based on the presumption that in order to certify an 
individual for practice in an occupation that requires licensure, licensure boards must have a record of 
individuals who pass or fail required licensure examinations. Whether they also record the individual's 
postsecondary institution and can aggregate individuals by those institutions is not a certainty. 
Similarly, it was conceivable that agencies that accredit either institutions or programs within 



'^Research ind Evatuation Assadettt, State Level MeasurBment of Peffonnanc« Qutcomei in PoMecwxtefv Education. Vo lumt 1: Executive Summnn-. An 
Oveivtew of State PBltcIa*. Contractor Repwt, US. Dcpattmem of Education. OfDco of Plaoniog, BaOg^ and BvaluatJoi*. 1990. 
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institutions might require student outcome data as part of the criteria for initial accreditation or for 
renewing accreditation. If this were the case, then accredited institutions or programs would have these 
data available. 



To get a sense of whether these suppositions were realistic, NCES conducted telephone interviews with 
licensure board personnel for five professions: Accounting, Cosmetology, Real Estate, Architecture, 
and Nursing. Although the limited number of licensing agencies is not representative of licensing 
agencies in the nation, it was felt that these interviews would provide some insights into whether there 
were consistent practices among licensing boards in their interactions with poslsecondary in.stitutions, 
both across areas of responsibility and across states. 

Of particular interest, was what feedback, if any, licensing boards provide to graduating institutions 
regarding the performance of their students on licensing examinations. Do they generally compute pass 
rates on an institution by institution basis? If so, how do they compute them? Do they report them 
to the institution or to the general public? If not, do they report individual student performance to the 
graduating in.stitution? Additional questions ascertained the level of concern with individu:>l privacy 
issues, and the difficulties involved in state licensure boards' providing pnss rate data to institutions. 

Accreditation in postsecondary education is a voluntary process of quality evaluation. In IW, there 
were fourteen institutional accrediting bodies recognized by the Department of Education as accrediting 
institutions of higher education. Nine of them were regional associations that accredit institutions and 
five of them were national associations that accredit types of institutions. There were an additional 100 
specialized agencies that accredit occupationally oriented postsecondary education program.s. In 
developing an accreditation process, accrediting organizations generally establish standards that mav 
mvolve measuring the outputs and/or outcomes of the program, even, perhaps, student outcomes that 
can be tied to participation in the institution's programs. 

NCES contacted a few institutional and program accrediting organizations to determine if accreditinu 
agencies, as a general rule, require student outcome statistics from Institutions or programs as part of 
their standards for accreditation and evaluation. 

C. An Assessment of the Current Status of Institution-Level Student Outcome Data 

1- Institutional Capabilitv for Renortinu Student Outcomes 

Results from the NCES Fast Response Survey of institutions indicate that 79 percent of all 
postsecondary institutions in the country can report a graduation/completion rate. Amonu 4-vear 
schools, 83 percent of public and 74 percent of private non-profit institutions can report a 
graduation/completion rate. Additionally 64 percent of public 2-year institutions and over 80 percent 
of for-profit schools can report this statistic (table III-l). 

Of the almost 8,000 .schools that can report this statistic, over 80 percent indicated thev could report 
graduation/completion rates by gender, attendance status, and final major field of study; about two- 
thirds could report by race/ethnicity, age, initial major field of study, and residency status; while over 
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one-half could report them by school or academic division within the institution and the student's 
general financial aid status. Table IIM presents these data by level and control of institution. While 
there is some variation among sectors, there are not many large disparities between a specific sector 
and the average for all institutions. 

These results indicate that most institutions can disclose or report graduation/completion rates and 
report them by students' personal characteristics (gender, race/ethnicity, age), and enrollment 
characteristics (major field of study, especially final major field of study, attendance status, and 
residency status). However, less than half of all institutions could report them by financial aid status 
or school or academic division within the school. 

In addition to the capability for reporting graduation/completion rates, the survey also explored 
institutional capability for reporting the other institution level student outcomes specified in section 103. 
Interestingly, 80 percent of all institutions reported being able to determine employment outcomes for 
graduates/completers. This percentage varied considerably by institutional sector, however (table III-2). 
About half (52 and 55 percent) of public and private non-profit 4-year institutions reported this 
capability. The majority of the institutions determined employment outcomes, primarily through sample 
surveys of graduates/completers (53 percent report**d using this .specific methodology). Nineteen 
percent of public institutions reported determining employment outcomes through state record systems. 

Fifty-eight percent of all institutions reported being able to determine if graduates/completers obtained 
necessary or appropriate licenses or certifications (table II1-3). However, this also varied by 
in.stitutional sector. Four-year institutions (public and private nonprofit) and 2-year and above 
proprietary institutions were generally less likely to have this capability than other types of in.stitutions 
(table 111-3). Twenty-eight percent of institutions that determined licensure or certification outcomes 
for graduates/completers surveyed a sample of graduates, and 51 percent obtained this information 
through state record systems. Surprisingly, the percentage of institutions that used state record systems 
did not vary appreciably between public and private institutional sectors. 

Section 103 of the SRK Act requires that institutions consider a student as a completer if the student 
transfers to another eligible institution for which the enrolling Institution provided substantial 
preparation. To gauge institutions' ability to include transfers in their graduation/completion statistics, 
institutions were asked if they could determine if a student had actually transferred from their school 
to another institution. Of the institutions that could report graduation/completion rates, only 15 percent 
said they could determine an actual transfer to another in-state institution; 14 percent said they could 
determine a transfer to another in-state private institution; and 12 percent reported they could determine 
a transfer to an out-of-state institution (table 111-4). Thus, of all statistics specified in section 103 of 
the SRK Act, transfers to another institution seemed to be the least available and most problematic for 
the vast majority of institutions. 
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Tabl0 IH-2.~P0rcen!age of postsecondary institutions that can determine employment outcomes for 
graduates/completers by method of collection and by control and level of institution: 
50 States and District of Columbia, 1991 
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86.4 
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06 


8.1 
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4.417 
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920 


48.9 


10.7 


9.6 


*This is the number of institutions In the 50 States and the District o^ Columbia that met the criteria for inclusion 



in the NCES postsecondary institutional universe in the fall of 1990. 
NOTES; Percentages by method of collection are based on institutions that can determine employment outcomes- 
Percentages do not add up to 100 because respondents were able to choose more than one option. 

SOURCE: U,S, Department of Education, National Center for Education Statistics. 

Fast Response Survey System^ Postsecondary Institutional Reporting Capability, FRSS 43. 1991 
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Table Percentage of postsecondary institutions that can determine If graduates/ 



completers obtained necessary or appropriate licenses or certifications, by 
method of collection and by control and level of institution: 50 States and 
and the District of Columbia, 1991 
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'This is the number of institutions in the 50 States and the District of Columbia that met the criteria 

for inclusion in the NCES postsecondary institutional universe in the fall of 1990. 
NOTES: Percentages by method of collection are based on institutions that can determine 



employment outcomes. Percentages do not add up to 100 because respondents 
were able to choose more than one option. 

SOURCE: U,S. Department of Education. National Center for Education Statistics, Fast Response 
Survey System, Postsecondary Institutional Reporting Capabitity. FRSS 43. 1991 
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Table HI-4.— Percentage of posisecondary Institutions able to determine completion/graduation 

rates and able to track transfer students to the transfer institution, 
by control and level of institution: 50 States and District of Columbia, 1991 
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.w .vfc VI \^viwnfuia inai inci itix? WiHtrrid lUT inLIUSH 

the NOES postsecondary institutional universe in the fall of 1990. 
NOTES: Percentages by type of institution transferred to are based on institutions that can 

report graduation/completion rates. Percentages do not add to 100 because respondents were able 
to choose more than one option. 

SOURCE: U.S. Department of Education, NaUonal Center for Education Statistics, 

Fast Response Survey System. Postsecondary Institutional Reporting Capability. FRSS 43, 1991 
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Results from the ACE survey of higher education institutions (an identifiable subset of postsecondnry 
institutions) concerning their assessment of student learning indicate that 72 percent of all non- 
specialized higher education institutions that serve undergraduate students assess the basic college-level 
skills of their students. Ninety percent of public 2-year institutions, 64 percent of public 4-year 
institutions, and 58 percent of all (2- and 4-year) private, non-profit schools reported having basic skills 
assessment. An additional 17 percent planned such an assessment. Thirty-five percent of the higher 
education institutions reported testing for knowledge in general education subjects or in a major field; 
24 percent reported a.ssessment of higher order skills in critical thinking, 32 percent in quantitative 
problem -solving, and 56 percent in writing. Significantly, another 30 to 40 percent planned higher 
order skills assessments.^* 

Among public institutions that are required to submit assessment data to a state agency (25 percent of 
all higher education institutions), almost 75 percent reported their assessment information was made 
public and 71 (18 percent of all higher education institutions) percent feel the assessment information 
is comparable across institutions.''' More detailed results and technical notes regarding this survey arc 
available in Appendix C. 

2. State Capability for Reporting Student Outcomes 

a. State Higher Education Agencies^ 

Thirty-nine state higher education agencies (including Puerto Rico) either have , are developing or are 
planning to have the capability to report graduation rates for postsecondary students. Of these, 19 
presently report graduation rates, 8 have a system in place and are developing the capability to report, 
and 12 are developing the capacity to within the next few years. 

The information collected and the analytic capabilities of the systems also vary from state to state. Of 
the 39 states with current or planned reporting capability, 28 states can or will soon be able to report 
graduation/completion rates across all in-state, public 4-year institutions. Twenty-two states can or will 
soon be able to report graduation/completion rates for the entire public state/system. Thirty-four states 
can or will report comparative rates by race/ethnicity, 33 by sex, 26 by age and attendance status (full- 
time, part-time), 28 by residency status, 21 by high school of origin, 16 by participation in remedial 
coursework, 1 1 by general financial aid status, 9 by athletic aid status, and 3 by income. 

The number of states reporting rates by field and by school/program within an institution is 27 and 6, 
respectively. Regarding whether and how states measure "other outcomes", only 5 states report post- 
graduation outcomes (such as employment) using follow-up surveys, and only 1 state uses state 
employment data. 



EI-Khnwas. Cimpus Trends , 1991. op. cit, p. 38. 
Nd.. p. 39. 

'tenth and Russell, op. cii. 
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b. State Vocational Education Oversight Agencies^^ 



Oversight of the 2-year and less-than-2-year vocational institutions (public community colleges pubMc 
vocational or technical schools, and for-profit vocational/technical schools) varies considerably from 
state to state. In only eight states are all three types of institutions the responsibility of ihe same 
agency. As a result, the level of involvement of state vocational education oversight agencies in 
collecting, maintaining, and/or reporting student outcome data is probably different for public sector 
schools (2- and less-than-2-year institutions) and for-profit vocational/technical schools. 

Of the outcomes specified in section 103(c) of the SRK Act, 46 community college and public 
vocational/technical school oversight organizations reported collecting graduation/completion data. 
These 46 state agencies represent about 90 percent of all public postsecondary institutions with 
vocational programs in the 50 states and the District of Columbia. However, only 34 of these 46 
organizations reported actually computing a graduation/completion rale. Thirty-three state agencies 
reported collecting placement data for public postsecondary school vocational completers. Twentv- 
.seven of these agencies, representing 46 percent of public 2.year and less-than-2-year schools collect 
data on training-related placement, although the methodology for collecting these data varies somewhat 
among states. Only 12 state agencies reported collecting licensure information. Therefore, only about 
one-third- of public schools with vocational programs would have these data available through a state 
agency. * 

Regarding other outcomes, 21 state agencies collect earnings information on public posisecondnrv 
vocational school completers and 13 measure gains in basic skills. These represent 37 and V percent 
of public 2- and Iess-than-2-year institutions, respectively. 

Twenty-nine state oversight organizations of for-profit vocational/technical schools collect completion 
data, 21 collect placement data, and 12 collect data on training-related employment. These agencies 
represent 80 percent, 62 percent, and 26 percent of the for-profit vocational/technical schools in the 
country. Agenc.es m two states representing about 11 percent of for-profit schools collect licensure 
mtormation. Only 19 state agencies compute a graduation/completion rate for students in for-profit 
vocational technical schools. Very few other student outcomes are considered for these tynU of 
institutions. Only three state agencies collect earnings infomiation for completers of for-profit 

r^Tl'^^! ' '"V^'r "^"'^ '^'^ "''^ ndvertising. Again, however, comparability 
is a problem because of definitional differences among the states. 

c. Licensure Boards 

k'^moi^c ''"'^ ^"J^' ""'^ '''''' ^"^ '"^'^'^"'-^l "^^"«'"S boards that were 

ZZ Z i ^/ff ^PP^o^ched reporting examination passage data to postsecondary institutions. 
Many of the differences among states were due primarily to differing priorities within the state. 
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Several states permit the reporting of a pass rate on state licensure exams by institution, but not by 
individual student, or by any characteristics that cnuid identify individual students, e.g., gender or race. 
Other states, such as Mississippi, allow reporting to institutions of the individual student's piuss/iail 
.status. In this situation, institutions could conceivably calculate pa.ss rates. 

With regard to the practices of individual licensing boards, some relied on testing services or national 
licensing boards to administer licensing tests and calculate institutional pass rates based on test takers. 
For example, a national board, the National Association of State Boards of Accountancy (NASBA), 
ai'nini.sters exams to prospective accountants, and calculates pass rates by institution, among other 
statistics. NASBA then makes this information available to the public in book form. 

In architecture, all of the nine state boards used the Educational Testing Services (ETS) examination 
that is administered through the National Council of Architectural Registration Board (NCARB). 
NCARB compiles augreiiate .statistics based on test results reported by ETS and reports the results by 
uraduating in.siitution back to the state. The state boards, on the other hand, do not necessarily report 
back to institutions within the state. When asked about thi.s, several boards cited limited resources. 
Others said that the NCARB took care of .statistics. 

■Hie reporting practices of state licensure boards of realtors vary considerably. The Realtor Licensing 
Examinalion'is g -nerally administered by a national contract agency. This agency provides statistics 
to both .state licensing boards and to institutions. State licensing boards receive aggregate pass/tail 
.statistics for each institution, but the institutions only receive their mean score and its standard 
deviation. Few state boards calculate or report institutional pass rates because of resource ct^nstrainls. 
One state does not require individuals to take courses in order to sit for the realtor's exam, and thus 
in.stitutional pass rates are not considered meaningful. Only one state surveyed calculated and provided 
pa.ss rates by in.stitution on the realtor examination. 

State licensure boards of cosmetologists are typically under the jurisdiction of the Department of 
Health, and they generally maintain institutional pass rates and report them back to institutions. 
Privacy was a concern, but it was overridden by the pervading belief among these boards that schools 
already know the grades and academic caliber of their studems and pass rates would not provide any 
new or confidential information. The states that do not report pass rates generally cited limited 
resources as a con.straint. State licen.sing boards for nurses are also typically under the jurisdiction ot 
the Department of Health. All state nursing licensure boards questioned .said that they report and 
maintain pass rates by institution. 

In summary, there are a significant number of different licensure examinations that are required of 
postsecondary education completers and there is tremendous variation in the way the different 
occupations/professions and states handle pass rate information. As a result, in multi-program 
postsecondary institutions, licensure pass rales would have to be obtained from multiple sources. 
Further, many states have state privacy laws that prevent boards from releasing information pertaining 
to an iridividual student. Some boards obtain relea.se statements from students, and thus are able to 
relea.se pa.ss rates by student. As a general rule, however, boards that report pa.ss rates by institution 
do .so in aggregate and thus are within the limits of state privacy laws. 
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Additionally, it would seem that boards that already calculate pass-rate statistics probably would be 
able to provide that information to schools. Boards that do not calculate the statistics will find it 
burdensome to do so, although those professions that have national coordinating councils or national 
licensing boards that administer the test will find It less burdensome to provide this information than 
individual boards within a state. 



d. Accrediting Agencies 

NCES contacts with a limited number of accrediting agencies revealed that most accrediting 
organizations are sympathetic to budget constraints and simply ask for "reasonable evidence" of 
positive student outcomes. However, an institution or program unable to provide such evidence might 
still be accredited if everything else were in order. Accrediting organizations tend to believe thai 
studem performance outcomes do not necessarily separate good institutions from bad institutions. A 
representative of a regional accrediting organization expressed the opinion that student outcome 
statistics for accrediting purposes were meaningless because they were not necessarily comparable 
across institutions. He felt this was especially true for regional accrediting agencies that accredit 
mstitutions in several states and standardization of these statistics in public in.stitutions across states 
is unlikely. Thus, the organization does not require institutions to collect or report outcome statistics 
as part of the accreditation process. 

Vocational and occupational program accrediting agencies were more likelv than institutional or 
professional program accrediting agencies to collect outcome statistics. The National Accreditinu 
Commission of Cosmetology Arts and Sciences, for example, has integrated an analysis of aulcoml 
measures imo the overall process of accreditation by collecting information on the completit>n 
placemem and licensure rates of students via an Annual Report, that is due from the school to the 
Commission on December 31 each year, and by requiring all schools to disclo.se these rates to 
prospective students. To calculate graduation rates, the Commission uses an approach that traces 
graduates backward to determine when they started rather than tracking beginning .students and 
placements are calculated as a percentage of completers. 

Similarly, the Commission on Occupational Education Standards of the Southern Association of 
Colleges and Schools has a .standard matrix that it requires institutions to complete as part of the 
accreditation process that includes outcome measures. However, neither of the.se two agencies require 
validation of the outcome data. Thus, the accuracy of such .statistics is unknown. 

D. Summary and Conclusions 

There seerns to be an unexpectedly large number of institutions that maintain and report .student 
graduation/completions rates, and employment statistics on their completers. Fewer but still a 
surprisingly high percemage of institutions determine licensure pass rates for their' completers 
Additionally, most state agencies collect graduation/completion data, and many can determine the 
graduation/completions rates of primarily public institutions in their state. States, on the other hand 
do not tend to collect employment-related outcomes, except for completers of public postsecondarv 
occupationally specific programs. k 
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The apparent availability of institution-level student outcomes suggest that nationwide reporting of 
some institution-level outcome statistics might be possible, whereas others would not be possible. The 
suggestions are summarized below. 

(1) Graduation/completion rates of undergraduates based on receipt of degree or on meeting 
all programmatic requirements for the degree would be possible. Additionally, these 
could probably be reported by static characteristics of the student such as gender, 
race/ethnicity, ag-% final major field of study, attendance status at time of completion, 
financial aid status, and residency status as well as by academic division within the 
school. 

(2) Determining transfers from the enrolling institution to another is not possible for the 
enrolling postsecondary institution. 

(3) Employment outcomes of graduates/completers (training-related employment, earnings) 
are also possible, especially for vocational/occupationiil programs in public institutions. 
These statistics are less available for public or private 4-year institutions. 

(4) Assessments of students' college level basic skills are possible at least for those 
undergraduates enrolled in 2- and 4-year higher education institutions. (These do not 
include the vast majority of private, for-profit institutions.) Assessment of higher order 
cognitive skills is not likely. 

(5) Passage rates on licensure/certification examinations would be possible for very specific 
fields and/or single program institutions. For institutions with .several diverse programs 
that require licensure or certification, an institution-level pass rate is not a meaningful 
statistic, and cannot be readily be determined. 

It must be reiterated and emphasized that the.se possibilities are based entirely on the current prevalence 
of student outcome stati :tics. They do not address the quality of the data that are currently available, 
nor do they address the utility of currently available institution-level student outcome data for any of 
the purpo.ses noted in the previous section. These very critical issues are considered in the following 
section. 
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SECTION IV 



PROBLEMS WITH INSTITUTION-LEVEL 
STUDENT OUTCOME REPORTING 

A. Introduction 

Determining what institution-level student outcome data are currently and generally available was a 
critical element in the process of examining the feasibility of reporting nationwide institution-level 
student outcome data. It provided some clues about what data might be feasible at a national level 
after some development has occurred. That is, it might be postulated that if some institutions are 
collecting and reporting particular data, then all institutions could collect and report those data. On the 
other hand, simply knowing that some number of in.stitutions and/or states have and even report some 
data, provides no indication of the quality of that data (i.e., its validity, consistency, or lack of bias). 
Nor does it provide a sense of the utility of the data for specific uses, or the co.st of collecting them. 

It is not possible within the scope of this report to evaluate the quality of the student outcome data that 
are maintained by individual institutions, or even by each of the states that report having these data. 
This would entail having detailed knowledge of the specific definitions as well as an understanding of 
the procedures and methodology used in collecting the data. It is possible, however, to get a sense of 
the quality of some types of outcome data oy reviewing past attempts to collect or regulate student 
outcome data at a national level. This is presented in part B of this section. In addition, data obtained 
from the NCES survey of p{)stsecondary institutions concerning definitions and methods used to report 
.student performance data and information from case studies of states' student performance data 
collection efforts will provide some insights into the problems involved. In terms of the utility of 
available student outcome data, institution-specific definitions and procedures that produce good quality 
data are adequate and appropriate for the individual institution's self-evaluation. They are probably 
adequate as well for most state and other institution needs for student performance data to determine 
accountability. 

For consumer protection purpo.ses, however, while it may be argued that some information is better 
than no information, non-comparable or incomplete infomiatlon is probably worse than no information 
at all because it could lead to inadvertently false choices. Thus, part C of'this .section will address, as 
directly as possible, the two characteristics that make student outcome data useful for consumer 
protection purposes: comparability and completeness. 

B. Previous Attempts to Collect Nationwide Student Outcome Data 

While efforts to collect institution-level student outcome data are extant in many of the states (see 
section III B), only three remotely related efforts at the national level were identified. These were the 
Vocation Education Data System (VEDS), and the attempts of the Federal Trade Commission, and then 
the Department of Education's (ED) Office of Po}»tsecondary Education to require institutional 
disclosure of student outcome information. All three of ihe.se efforts were reviewed for this report by 
ab.stracting internal documents and conducting interviews with knowledgeable persons concerning: (a) 




the implementation, results, and subsequent suspension of VEDS; (b) the Federal Trade Commission's 
attempts to require postsecondary institutions that operate for profit (i.e., proprietary institutions) to 
disclose graduation rates and rates of training-related placements for completers; and (c) the publication 
and suspension of the ED's Track Record Disclosure regulations. 

1. The Vocational Education Data System fVEDS'> 

The Education Amendments of 1976 mandated the development, implementation, and operation of a 
national vocational education data reporting and accounting system (P.L. 94-482, section 161). In 
response to this requirement, the Vocational Education Data System (VEDS) was developed and 
implemented by NCES. The 1978-79 school year was the first year of data collection. VEDS was to 
include data on enrollments in occupationally specific programs, information on programs offered, 
numbers of program completers and leavers, and vocational education staff, facilities, and expenditures. 
The system was also to include the results of mandated state evaluations of vocational programs, 
reporting the extent to which program complete!^ and leavers- 

"i. find employment in occupations related to their training, and 

ii. are consideration by their employers to be well trained and prepared for employment" 
(P.L. 94-482, section 112). 

VEDS was developed as a state-level reporting system and no individual institutions were identified 
through VEDS at the national level. Rather, states had the respon.sibility for aggregating data 
concerning vocational education programs and students from all institutions that received federal 
vocational education program dollars in the state. States could collect the data at a state level through 
a state-level student unit record system or aggregate individual institutional reports for submission to 
NCES. 

State agencies responsible for VEDS data felt their most significant problems resulted from: (1) 
attempts to link individual student data that were collected at different points in time or at different 
administrative levels; and (2) reporting student data by detailed program characteristics (e.g., 
instructional setting, legislative purpose). The problems with linking data were reported to be less 
severe in states that had student unit record systems at the state level. Even these states, however, had 
problems with linking completer/leaver and employer follow-up data to student enrollment data."^ 

To obtain data that would satisfy section 112 of the Education Amendments of 1976, states or 
institutions surveyed completers through a mail questionnaire that was generally unique to each state 
or even each institution. The questionnaire sought information on the completer's employment status, 
educational status, and/or milhary status. For employed completers, it attempted to ascertain if their 
job was related to their training. States aggregated the data from this survey for submission to NCES. 
For respondents to the student survey who indicated a related job placement, states or institutions would 
survey employers to obtain information on employer satisfaction with the student's on-the-job 
performance. 
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The VEDS experience concerning completer and employer follow-up data (i.e., reporting the extent to 
which program completers find employment in occupations related to their training, and are considered 
by their employers to be well trained and prepared for employment) provides valuable information 
about the feasibility of institutions' disclosing training-related employment outcomes of 
graduates/completers. It also addresses whether employer satisfaction with completers/graduates might 
be one of the ''other" feasible outcome measures mentioned in section 103 of the SRK Act. 

In an attempt to gain insights into the quality of the data reported on the 1978-79 VEDS reports of 
student outcomes and employer satisfaction, NCES determined the response rates of both the student 
follow-up and the employer follow-up surveys.^' Response rates are good preliminary indicators of 
data quality, for without adequate levels of response, other aspects of data quality are, for the most part, 
moot. 



Three elements went into determining the response rate at the national level in the VEDS follow-up 
surveys. The first element was the number of entities (states) that submitted follow-up data to NCES 
relative to the total number that had programs. The number of entities that had programs was 
determined from the 1978-79 enrollment report, and the number of emities that submitted student 
follow-up surveys was determined from the follow-up reports. Eighty-two percent of the 45 entities 
with postsecondary vocational completers reported completer follow-up data. 

The second element in determining the response rate in the VEDS follow-up surveys was the respon.se 
rate of completers to the follow-up questionnaire sent by the states or individual institutions to 
determine the completer's employment status. This rate was estimated, using summary data provided 
by the states, by calculating the ratio of the total number of students who were reported in each of the 
five employment status categories that suggested a response (employed in a related field, employed in 
an unrelated field, pursuing additional education, unemployed, not in labor force) to the total number 
of completers reported in the follow-up. The response rate for postsecondary completers was 71 
percent. When stale non-respon.se was considered in conjunction with student non-response in 
determining effective response rates for postsecondary completers, the effective response rate declined 
to 58 percent. 

The third element for determining the response rate in VEDS follow-ups was the response rate of the 
employer follow-up. As noted, the employer follow-up was conducted only for respondents who 
mdicated they were employed in a field related to training. In addition, it was possible to survey 
employers only if students reported the name and address of their employers on their questionnaire'. 
Thu.s, there was a pervasive question as to what constituted the universe for the employer follow-up. 
Was it all students in a related employment, or was it only those students in a related employment that 
reported employer name and address? 

Detecting the percentage of respondents that supplied an employer name and address was not possible 
for the 1978-79 collection year. In the 1979-80 collection year, states were instructed to provide the 
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number of completers and leavers that reported an employer name and address. For those 10 states 
that reported these numbers, an estimated 71 percent of students In a related employment supplied an 
employer name and address. Using the data from these 10 states, a rough estimate of employer 
response rate was found to be 55 percent. 

Using these rough estimates, it was possible to determine the effective response rate on the employer 
follow-up by using the effective response rate for postsecondary completers (58 percent), the proportion 
of students that supplied an employer name and address (71 percent), and the response rate of 
employers (55 percent). Combining these factors yielded an effective response rate for employers of 
23 percent, 

2. The Federal Trade Commissioner Regulations on Student Outcomes 

For many years the Federal Trade Commission (FTC) contemplated requiring proprietary vocational 
and home study schools to disclose graduation and job placement information to prospective students. 
On December 28, 1978, the FTC issued a final regulation requiring these schools to do so.^ Tliis 
regulation, however, never became effective because the Second Circuit of the U.S. Court of Appeals 
set it aside. Although FTC staff considered issuing a new regulation designed to meet the Court's 
concerns, the FTC decided against it. 

The regulation was designed in part to address a concern that prospective students were not receiving 
useful information about student outcomes in the programs in which they were considering enrolling. 
Schools were required to disclose information, by program, about graduates from the prior academic 
year on a specific form. Schools were allowed to deviate from using this form only under 
circumstances specifically cited in the regulation. 

The FTC regulation required schools to disclose the following information regarding program 
completion: 

o the number of students enrolled; 

o the number and percentage of students who graduated; and 

o the number and percentage of students who failed to complete.*' 

In addition, this regulation required schools to disclose the following information about placement and 
earnings of graduates 4 months after successful completion of the program if the school promoted the 
program by making job placement or earnings claims: 
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o the number of students who graduated; 

0 Jhe^umber and percentage of graduates who got jobs in the field for which the school trained 

° Sand °' ^"""^ "'ff"™' -'-^ increments of 

o the nutnber and percentage of students who refused to provide salary or earnings infomation. 
JiSlto'" '"f"™"""" - 'he same mailing with the 

..rii-r:;^:eS:rrd^^^^^ 

mnT^i!l'"\""""' ™y "e contained in the same envelope. The school 

may not show how many students could not be contacted or did not respond to fob 

Tur momhrr'"- h" ""•"^ "'^'"^^ ""^ ""'"^ °' " «k jo ^ within 

mplZ n com,Wd""b" "^"""'^ P"" ™Pl"yn,ent, .4 f- 

einployment, eontmued schooling, or other reasons. Proof in the record show, thnf 

hi fJr " ""'P'oy™'"' »"«ess rate of those who responded to 

he school s mqu,ry was 54%. or 80% if those who became self-employed we e 

ro,.H -.H '^'"«'="' ''^^If «ith the same Zem'c cave t 

Sngs - P-«'g' of -"-g additional informatir in S 

TTiat caveat is a statement schools were allowed to use to explain their placement rale: 

1 iohf — ■ , " ^^'"^l' »f ""f ^'"I'^nts 10 see if they 

got jobs. So, our placement percentage might be understated.^ ^ 

^.^Z ^^^Z^:s°'^^-r;r^'^ ^'-e for detemng 

that they can be^ccurlly" " " P'''^''*^ *°P<»« nnd 
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The opinion of the Second Circuit Court of Appeals, however, was not unanimous. The dissenting 
opinion supported the FTC's decision to require the disclosure of job placement information m the 
specified format, stating, in part: 

The majority refers to one instance where the Commission's Rule requires a school to 
report, truthfully, that the known job piacemem rate is 5.8%, whereas what the majority 
calls "the true employment success rate of those who responded" was 54%. There are 
two reasons why it is highly misleading to call this 54% a "true employment success 
rate " First as the majority acknowledges, but as schools rarely do, this rate is based 
only on those who responded to the school's inquiry. The majority's example is a 
television servicing school that had 2,271 graduates, of whom only 929 responded to the 
job-placement inquiry. Neither the majority, the school, nor prospective students who 
may have read the school's advertisement have any reason to think that the placement 
rate for the more than half of the graduates who did not respond is anywhere near as 
high as for those who did. 

Secondly, as the majority fails to acknowledge, the 54% success percentage is not based 
on all the 929 graduates who responded, but only on the 244 of these (26%) who 
reported that they were seeking full- or part-time jobs. Again, neither the majority, nor 
the school, nor prospective students have any way of knowing how many of the 685 
students not seeking jobs dropped out of the labor market because of the difficulty of 
finding a job or because the course left them woefully unprepared to handle one." 

Both opinions reveal the difficulty of determining how to calculate job piacemem rates. Assuming that 
the disclosure of these rates affect consumer behavior, how job placement rates are detined could have 
a great deal of impact on consumer behavior. The job placement rate of a single program could vary 
from as low as 5.8 percent to as high as 80 percent based on how it was defined. 

3. Experience With Track Record Disclo sure Provisions 

I he Department of Education's O^nsumer Disclosure Provisions [34 CFR 668.44 (c)-(f)] required 2- 
year community colleges, private for-profit vocational schools, and other schools with sub-baccalaureate 
vocational programs to disclose to prospective students the completion rates, piacemem rates, and state 
licensing exam pass rates for occupationally-specific vocational programs. The problems schools 
experienced in complying with these regulations provide additional insights into institutional collection 
and reporting of student outcome statistics. 

Results from a survey of associate degree-granting institutions concerning these regulations'" indicated 
that only 50 percent of respondents reported their institutions to be in compliance with them. Further, 
the substantial variation in the way respondents imerpreted the regulations led to estimates that the true 
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compliance rate was closer to 30 percent. (In interpreting the results of this study, it must be noted 
that only 39 percent of the institutions contacted for participation responded to the survey.^^) 

TTie major problems with the regulations centered around the fairness of requiring these disclosures 
only from schools with sub-baccalaureate vocational programs; the difficulties with, and cost of 
compliance; the lack of definitional specificity for required statistics;'-^ the inherent difficulty of 
collecting reliable and meaningful placement information; the accuracy of the statistics generated; and 
the potential usefulness of the information. 

fnZ'f Ik' T^^'' difficulties identified included: the burden and expense of extracting and 
coinpiling the information from manually maintained files; schools that did not maintain the required 
Irnc!^^?" ^ r f "^P'y^!^*^ regulations in the time required; and institutions with several 
programs covered by the regulations found it impossible to track completion and placement rates for 
all of their programs, especially if their record systems were manually maintained. Furthermore the 
prob ems of reporting statistics by specific program were amplified for two reasons. First, manv 
institutions had difficulty determining exactly what constituted enrollment in a specific program of 
study; and second, students changed from one program to another making it difficult to track 
performance in the original course of study. 

^Hn^l^'r ""-'^1^1 '^'"'^ ^'""'^ ^'""^"'"^'^ Regulations was estimated at around $8,0(K) 

per institution. TTiis raised the question of the prudence of committing scarce resources to provide 
consumer information unless it can be determined that such information is accurate, fair, and useful 
to consumers. u.^wiu* 

The open-ended approach to definitions and procedures taken in the regulations was disconcerting to 
institutions that were attempting to make good faith efforts to comply. Moreover, it meant Th-,t 
vThUitv not being provided with comparable statistics. To illustrate the extent of the 

cohort!!"^ clefmuions ot a cohort were identified among 56 institutions that tracked 

Difficulties also surfaced among schools attempting to collect training-related placement data The 
most popular method of determining placement involved surveys of graduates. These suffered from 
ehtrd'^Ic^^^^^^^^^ '"'^ '''' determining exactly what constituted a trnining- 

n hutions^ """" ' ' P'"'"'' differentially applied nmonu 
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The regulations were suspended by the Student Right-to-Know Act, ahhough schools that publicly 
make a claim about job placement rates must still abide by other disclosure provisions in the Higher 
Education Act (section 487(a)8). 

C. Other Identified Problems 

In section III it was reported that a significant number of institutions have student outcome statistics 
available, especially graduation/completion rates. Having outcome statistics available, however, does 
not necessarily imply that the data meet the most important criteria for consumer protection use -- that 
is, it does not imply they are comparable among institutions or institutional sectors. 

To address the issue of comparability, several questions were asked of institutions in the NCES Fast 
Response Survey concerning the methodology institutions use in determining graduation/completion 
rates. Responses to these questions provide national estimates of the incidence of relatively comparable 
procedures and definitions among institutions.^ How these procedures are actually implemented, 
however, cannot be surmised. 

There appears to be some degree of similarity in the methods used for determining a 
graduation/completion rate. Seventy-nine percent of all postsecondary institutions could report a 
graduation/completion rate. Of these, about 71 percent (56 percent of all institutions) report following 
an entering class from initial enrollment to graduation/completion to determine their rate. However, less 
than one-half (46 percent) of the 64 percent of the public 2-year institutions that report 
graduation/completion rates seem to use this methodology. Public 2.year and less-than-2-year schools 
were more likely to retrospectively trace back through records of the members of a graduating class 
to determine graduate rates than were other types of institutions. 

About 75 percent of postsecondary in.stitutions reported having a student tracking systems. Of these, 
about one-half (54 percent) could calculate a graduation/completion rate for a cohort of .student.s (where 
a cohort is defined as a group of students identified by some common characteristics such as tlrst year 
enrollment). This translates to only about 41 percent of all institutions that could implement a cohort 
methodology. 

In schools where use of a cohort methodology is employed, however, there seemed to be a substantial 
degree of similarity in the definition of the initial cohort to track. Over 85 percent (or 34 percent 
overall) included first-time, first-year, full-time, degree/certificate seeking students in their cohort for 
calculating a graduation/completion rate and 62 percent (or 25 percent of all institution:;) included tirst- 
time, part-time students in their cohort. Surprisingly, 42 percent (or 17 percent overall) included any 
student registered for credit in their cohort, and 1 1 percent (or 5 percent overall) included non-degree- 
seeking, non-credit students. 

However, these similarities seem to break down in the process of tracking a cohort. For example, there 
was significant variation among institutional sectors in the percent of institutions that add transfers into 
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the institution to an already established cohort.^' Tliey ranged from a low of 17 percent of public 4- 
year institutions to a high of 84 percent of public less-than-2-year institutions. 

Additionally, institutions dropped students from an established cohort for a variety of reasons that 
differed for the different institutional sectors. For example, about 24 percent of all institutions reported 
dropping a student from an established cohort if the student changed from degree-seeking to non-degree 
seeking status. However, 40 percent of private non-profit institutions report dropping these students 
and over 30 percent of all 2-year institutions reported dropping these students. Institutions were even 
more likely to drop transfers out or stop-outs from their established cohorts (51 percent reported doing 
this), although reports of this type of action ranged from 21 percent of public 4-year institutions to 79 
percent of private, non-profit 2- and less-than.2-year institutions. 

Even the definition of a graduate/completer varied somewhat among institutions. Almost all institutions 
considered the recipient of a degree or certificate as a graduate/completer. Seventeen percent of the 
institutions considered a student who transferred to another institution as a completer although over 
25 percent of public sub-baccalaureate schools included transfers out of the institution as completers. 

Thus, while it would appear thnt there is .some comparability in methodology u.sed by po.slsccondary 
in.siitutions to determine a gradu-.tion/completion rate (most track students through a cohort 
methodology), the rates themselves are probably not a.s comparable as the similarities in methodology 
would .suggest. This is primarily due to real differences among institutional sectors in terms of who 
is included in establishing a cohort, who is added to an established cohort, and who is dropped from 
the cohort. 

Differences in definitions among institutional sectors even within a single state were cited as being 
extremely problematic in interviews with representatives of state vocational oversight agcncies.-^^ As 
the summary of this report emphasizes, one of the issues that arises from a division of oversight 
responsibilities among state agencies is variability in the types of information required of the schools, 
as well as in the definitions of various performance measures. This means that individual institutions 
are mo.st probably reporting different sets of information and using differinu definitions and criteria for 
outcomes of interest. 

This appraisal was confirmed by REA's survey of postsecondary vocational oversight agencies in the 
50 states. As the survey results presented in the previous section clearly indicate, \l would seem 
that state vocational education oversight agencies have a substantial amount of data that are related to 
student outcomes. However, the comparability of these data among states is questionable at best. For 
example, 38 states define a completer as one who completes all program requirements and is awarded 
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a degree, diploma, or cerlitlcate. However, 13 of them define completion as receipt of an AA degree 
only. Nine states define a completer as a full-time or part-time student who completes all phases, 
planned course sequences, competencies, etc. of a defined program. 

Similarly, as reported in the previous section, 27 of these agencies, representing 46 percent of public 
2-year and less-than-2-year schools, collect data on training-related placement. However, the 
methodology for collecting these data varied considerably among the states. Specifically, 23 states 
allow former students to make the determination of relatedness, 5 allow schools to do so, 2 allow the 
state agency to make the determination, and 1 allows employers to make that determination. 
Additionally, 24 state agencies defined placement as "program completed and employed in training 
related area;" 12 states explicitly include those who join the military or transfer to a four year college 
as being placed, and 7 define placement as "program completed and employed in any job." 

D. Summary and Conclusions 

Tlie problems identified in nationwide reporting of student level outcome information at a sub-national 
level (state or in.stitution) may be summarized as follows: 

(1) Use of surveys of graduates to a.ssess po.sl-completion outcomes (training-related 
employment, earnings, licensure) are generally unreliable. They may be biased as well 
since response rates to these surveys are fairly low and there is little information on the 
characteristics of non-respondents. 

(2) Definitions of critical terms are incon.sistent among institutions and institutional sectors 
nationally and within states that have different reporting requirements for different 
institutional .sectors. This seems to be the case for almost all student outcomes statistics. 

(3) From the ACE survey concerning higher education assessment, only 18 percent of all 
institutions felt their tests were comparable to tho.se of other institutions and these were 
.schools that were required to report assessment results to a state oversight agency. 
Moreover, it is not pos.sible to determine the extent of the comparability (e.g., only peer 
institutions within the state, all other public in.stitutions in the state, cro.ss-state peer 
in.stitutions). 

The prevalence of student outcome statistics suggests widespread interest in these data and good faith 
efforts to collect and maintain them for institutional or state reporting. However, just based on the very 
broad-brush view of the problems that were presented here, it would be reasonable to conclude that, 
at the present time, no nationwide student outcome data reported (or disclosed) by institutions or at an 
institutional level would be useful for the consumer protection purposes implied in the Student-Right- 
to-Know Act. Based on the quality of these statistics, none of them, including graduation/completion 
rates, are feasible. 

The Student-Right-to-Know Act, however, does not require disclosure of the.se statistics based on 
current data or reporting capability. Rather, the first disclosure (and reporting) requirements are 
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required in July 1993. This provides some time for effecting clianges in definitions and methodologies 
that will help create a more comparable and complete set of institution-level student outcomes statistics 
at a national level. 

With this in mind. Section V of this report presents the results of discussions with the postsecondnry 
community, expert thinking in the area, and a model of state-level data collection that can be used to 
describe some long-term goals in the reporting of institution-level student outcome statistics. 
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SECTION V 



WHAT IS NEEDED 

A. Introduction 

As Section IV indicates, virtually no currently available institution-level student outcome statistics 
would provide meaningful and useful nationwide data. However, it is possible to bring some order to 
the collection and reporting of the data if definitional consistency and standardized data collection 
procedures and methodologies are established at a national level and implemented at the institutional 
and/or state level. In fact, personnel from state postsecondary vocational oversight agencies advocated 
embarking on this process"^ and this process was initiated by special working groups comprised of 
representatives of broad segments of the postsecondary education community. This section identifies 
those aspects of student outcome statistics that require definitional specificity and describes data 
collection procedures and methodologies that could introduce some assurance of quality and 
comparability into the nationwide reporting of institution-level student outcomes. 

In approaching this task, each of the outcome statistics is treated separately since necessary definitions 
and "best practice" in collecting them would be different. However, just describing how good quality, 
comparable national outcome statistics could be achieved certainly does not guarantee their "feasibility." 
Tlicre is a huge gap between identifying good and appropriate practice and having that practice 
instituted on a universal basis. That is, there are additional steps that would have to be taken to 
facilitate the adoption of this best practice including the investment of resources (personnel, funds and 
sufficient time) if reporting of any nationwide institution-level student outcome data is to be useful for 
consumer protection or any other comparative purpose. 

For each outcome statistic considered, the general parameters for collection and reporting will be 
suggested. These parameters include critical elements that require consistent definitions, collection and 
reporting procedures, and populations of interest. They were developed by combining expert opinion, 
current institutional practice, and detailed studies of in-place systems that have demonstrated 
effectiveness and potential portability. In some instances this process identified specific definitions or 
methodologies and these are included. In other instances, it led to more than one alternative for the 
same parameter. In these cases, the different alternatives are presented, although a rationale is provided 
for possibly choosing a single "best" alternative. 

As this approach evolves it will become apparent that there are distinctions between those statistics that 
an individual institution could collect and maintain and those that must be developed and maintained 
at a state or po.ssibly even a national level. Also as a result of this discussion, it will become evident 
that there are certain outcome statistics that defy consistent and comparable reporting because of the 
diversity of postsecondary education in the United States. 
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B. Graduation/completion rates 



Conceptually, a graduation/completion rate is the percent of students who complete a postsecondarv 
program relative to the number who start the program. With this in mind, operational definitions of 
graduation/completion and postsecondary program must be established; students who start and students 
who complete must be specified; and the methodology used to determine the rate must be identified. 

For graduation/completion rates, the methodology used to collect the data and the procedures used to 
determme the rate are the most critical elements. They dictate the populations of interest and thev 
provide boundaries for the needed definitions. 



1. Determining graduation rates 



Among all groups consulted, there was a broad consensus that a forward-looking cohort methodolouv 
was the only appropriate method for producing a valid and meaningful institutional graduation rnte.^' 

Employing this methodology entails identifying all students who start a progr.im at one pmni in time 
e.g., a specified term) as members of a distinct "cohort" to be tracked. At subsequent time inlervnis 
possibly each subsequent term), enrollment information for each member of the cohort would be .uklcd 
to the cohort record as appropriate, until the end of a predefined tracking period is reached. 

For example, a cohort could be established each fall term, or in October of the school year, if academic 
th™ ohort"^' ^"^'''^"^"^ enrollment information could be added each fall for members of 

The population of interest would be individuals who start a postsecondary program. That is it would 
be comprised of those individuals who enroll in a postsecondary institution tor the first time in the fall 
t would include, at a mm.mum. tho.se enrolled for the first time in any po.stsecondnry institution (i c ' 
those who enr^rf TrT^"' '"^ Postsecondary institution). Alternatively, it could also inLc 
those who enroll for the first time m a given institution. This latter group would be comprised 
primarily of individuals who transferred into the institution from another postsecondary ins tmk " 
Advice on this issue has been somewhat conflicting.^ but there was a strong belief among som^^oup^ 



'^T^is w.. expa-sscd de.arly in th. f,., wording on imputing the SRK Ac. in Mnrth. 1<.;1. A ..mm.^ of ,hi. ...M, is 
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that individuals who transfer into an institution also should be aware of their completion possibilities. 
This was considered as particularly important for student athletes who transfer trom a 2-year to a 4- 
year college and have continued eligibility for intercollegiate athletics. If cohorts were established for 
beginning postsecondary students and for transfers into the institution, there was no question that they 
should be considered as separate cohorts and tracked and reported separately. 

Section 103 of the SRK Act requires that graduation/completion rates be determined only for firsi-time. 
full-time degree-seeking undergraduates. Translating this requirement into the parameters tor 
identifying who should be included in a cohort eliminates undergraduates who enroll for the first time 
on a part-time basis. This has the effect of neglecting almost 30 percent of students who enroll for the 
first time in any postsecondary institution.'" Furthermore, while there was widespread agreement that 
a cohort should be limited to students who are seeking a degree or other formal award, there was little 
agreement on limiting a cohort to full-time students only. Moreover, about 62 percent of institutions 
that currently use a cohort methodology to determine graduation/completion rates include part-time 
students in their cohorts. Even more significantly, students frequently change from full- to part-time 
status between enrollment periods or even within a single enrollment period. Therefore, the inclusion 
in a given cohort of students who enter for the first time as part-time students should be seriously 
considered. 

An alternative to including both full- and part-time students in the same cohort is to eslablish iwo 
separate cohorts of entering students-one comprised of full-time students and the other comprised ol 
part-time students. Each cohort would be tracked separately for the same specified period ot time and 
separate graduation rates could be reported. 

Two terms remain to be operationally defined""degree-seeking" and "graduation/completion." For the 
purpose of defining a cohort, degree-seeking would be enrollment in any course that can be used tor 
credit toward a degree or other formal award, regardless of whether the student has declared a formal 
intent to seek a degree. If first-time, part-timt- sftulents were included in the cohort, however, their 
intent to seek a degree should be formally declared. 

Graduation/completion would be receipt of any degree or formal award, regardless of the degree ot 
award the student was seeking on entrance, although there was some concern that this detlnition could 
result in the proliferation of certificate programs at 2-year schools and sub-baccalaureate programs at 
4-vear institutions?. 

Using the methodology outlined, stati.stics on both persistence (re-enrollment of members of the cohort 
in each succeeding fall) as well graduation rates could be generated. However, it has been suggested 
that both statistics (persistence rate and graduation rate) be compiled on the basis of enrollment at a 
single institution regardless of whether or not .students subsequently re-enrolled elsewhere. This is 
consistent with current best practice in student tracking and yields stralghttonvard. easily-interprctable 
.statistics. 



'*^-n«!. is bnst-d on 1<)S7.SS hlfthiT educniion enrollment dita fnw the Integnrted PosLwomiirj" lyucnlion r).it,i Sysicm {li'l-J5S) v. prewnuii in tiw Sbtc 
MtchtT r-ituc.->iii>n rn>r)lw. Third f'tiiiion. \)S>. IX-p.irtnwm of lidufatiim. 19<>1. 
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Students who withdraw from a given institution in order to transfer to another institution should not 
be removed from the starting cohort. Rather, the proportion of students in the cohort successfully 
transferring to another postsecondary institution should be separately ascertained and reported fin this 
secuon, transfer to another institution [higher level] is considered a separate outcome from graduation 
since It requires an entirely different set of procedures for consistent, unbiased, and comparable 
reporting among institutions. Procedures for determining students' transfer status will be indicated in 
a separate section.) 

Because the methodology suggested utilizes a cohort, two kinds of data elements are required. First 
descriptive information is needed indicating the student's status at time of entry; typically, this includes 
demographic and other background information necessary for reporting completion rates for different 
studen populations. Secondly, limited performance information is needed on a term-to-term basis- 
typically this information is extracted for each student in the cohort and used to update that student's 
record for the term in question. 

2. Reporting graduation rntes 
Reporting from the proposed system could include three major performance statistics, reported annually 



percent completed - This statistic represents the proportion of the identified starling 
cohort that had completed a degree or certificate up to and including the point at which 
the report is generated. 



percent persisting - This statistic represents the proportion of the identified starting 
cohort that re^nrolled at the institution in pursuit of a degree or other formal award at 
the time of reporting. This statistic would provide valuable contextual Information for 
interpreting percent completed. In addition, it would enable institutions to report a 
meaningful progress statistic in the first year(s) of a cohort's history, rather than waiting 
to report until sufficient time has elapsed for substantial numbers to complete the 
program. ' 



time to completion - This statistic represents, for those who completed, the aven.oe 
amount of time (e.g., number of years) required to complete the program. It also wo Jid 
provide contextual information of interest to consumers and would be valuable to the 
institution and state. 

All three statistics could be reported annually, probably in the fall, if records were updated as of this 
time period. They would include the percent completing at the point of "150% of normal" time to 
program completion.- All three could and .should be reported by static student variables a.s Those 
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specified such as race/ethnicity and gender and others identified in section 104 of the SRK Act. They 
can be reported since they are fixed once a cohort is identified and they should be reported because 
they would make graduation/completion rates more meaningful to potential students. 

About 41 percent of all postsecondary institutions reported having a record system that was able to 
track cohorts of students.-*^ Most other postsecondary institutions probably possess the base data 
necessary to develop the methodology indicated here and could construct or adapt a free-standing 
student tracking system from their base data systems. Costs to establish such a system tailored to the 
requirements of a particular institution initially average approximately $25,000. Systems or 
consortia of institutions can reduce these unit costs substantially using a common methodology. 
Nevertheless, they will incur costs in extracting the data from their local records system and converting 
it into the standard formats required by commonly-developed tracking software. Here, per-institulion 
costs of approximately $10,000 can be expected. 

Institutional costs of developing a student cohort tracking system can be reduced substantially if slate 
or system-wide unit-record data are used to centrally prepare completion statistics for each institution. 
Currently, several states have this capability in place, and additional states plan such systems in the 
near future. Most current state systems cover only public institutions, and sometimes contain only a 
subset of public institutions in the stale. However, with sufficient incentive and assistance the coverage 
of existing state systems could be extended to include private sector institutions and more states could 
develop these systems. Because of the large number of student records involved, costs would be higher 
than for a comparable single-institution installation, although the per-student cost would be less. A 
good estimate would be approximately $50,000 for a state of moderate size. 

For any of these options, time is required to develop the required system development, and for an 
established cohort to "mature" to the point of program completion. It is unlikely, therefore, that a 
significant number of institutions would be able to meaningfully report graduation rates using best 
practice methodologies before about 1995, 

3. Completion Rates bv Field of Studv or Academic Division 

Tracking a cohort by any variable that could change frequently during the tracking period is difncult. 
This, however, is the situation with respect to field of study or academic division. Additionally, the 
point at which major field of study must be declared by students varies considerably among institutions 
and programs, and oftentimes students simultaneously enroll in more than one field. All of these 
situations are especially prevalent in higher education institutions. There may be exceptions to these 
patterns for certain professional, occupational, and technical fields in which students enroll in single- 
program institutions or are admitted directly to particular programs and tend to pursue these programs 
in isolation. Reporting graduation/completion rates by these limited fields is not reasonable to 



"*NCl:S Fast Response Suney of Instiluiional Capi^iHty. table C-UI, appendix C. 

*'aH cosi estimates wsre provided by P. E^^^tt and D, Jones of the Hitlcmal Center for tligher Education Managemem Systems (NCIiriV^) aixi are ba^»d 
on their experience in developing inslirutional and sute student cohort tracking sysletns. 
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advorate, however, since such limited and selective reporting could be considered "unfair" and indeed 
would not be very useful from a comparability perspective ' 

sirridlic T"""- '°™p'"i°" by field 

ot study or acadeinic division. One measure would use the cohort methodology d«cribed earlier 
d^fsf ra^iHltr Th'' intended 'r^arCo^aca mi : 

prent of rdenN IT • T""" """^ ""'"'y- " '""icalc the 

percem of students who intended to major in a given field who actually completed in that field 

l~^s^7srur„r'^^^ ■ t-^c m:!'h':;/h i 

woum be to. (1) sort all students completing programs in a given year into their resoective fields r,f 
irst term of academic history; and (3) average these times over fields of study to yield a tvnicnl t me 
Sitfltrr^Xr -'^'^ --^-'^"^^ — ver, b/ca Je of .iSaX: 

C Student FoUow-up Statistics 

»mprers^Tntfei^ tfj? '"T'"'"" f '"""""'^ ""=>' '^'^ i"''"'"""" C--"!'" 
kT' f'='!:/"'P-<""s, or drop-outs) is to conduct a follow-up survey-either by telephone or 
through the mail. Fifty-three percent of all postseccndary institutions repor ed using this „ ,W 
oc uo. IT ^'h' "f sraduates/complete.. Additionally, many sta.esturr m y re ui 

3 col'! , "P"" J"''-P'"^'--'"en> ^'ntistics on this basis and some state twer 

agencies conduct such follow up surveys of completers of their institutions As wis -imolv 
demonstrated by the description of the Vocational Educational Data System (VEDS),' how ver ?es'Z e 
nttes using th.s method are generally inadequate, and inadequate response ra es Xn resu I n 
inc^istent. non-comparable, and biased data. Adequate res^nse rates T exTem? di ftku 1 u 
oton because of the inability to locate fonner students. Furthermore, once studentrail l„ " ed ,h" 
costs of intensive non-response follow-up are high. 

Even if an adequate response rate were made a standard for reporting outcome statistics ihis wn„l,i 

rs ituHo r ' "T""' "■"''^ ■"''""dology. MLing'suchTs, n ^w Ul f 

institutions (or states) to utilize significant resources. (NCES follow-up surveys of 4 ve r L, , e 

rp^^d^e^'ifA^r r """" ' p-nLrhTpitiiy ; 

oro^edore ihT, t H ^ response rales together with the .statistics themselves is a 

mesof commr, ""'""^ '^''•'""P'e. "«= Program rev ew 

Stan V knowir h " occupational/technical programs in Florida and Maryland). However 
simply knowing what proportion of students responded does not even suasest the consi^er.'J! ; 
representativeness of the data, particularly if the rate is low ®^ consistency or 



Ewell p. and Jooes. D. op. eit. 



41 



GO 



ERIC 



1. Employment of graduat es/completers 



An alternative methodology for determining employment rates of graduates/completers «vaUable 
the state level through electronically linking files containing graduates/completers wuh available 
Unemployment Insurance (UI) Wage-Record Data. 

A study of this approach in tracking completers of Job Training Partnership Act (JTPA) programs 
concluded that "obtaining post-program information from state UI systems is not only a viable option 
it is far more cost-effective than the current practice of gathering this information through contact w. h 
participants."^ The report goes on to indicate that there are no "insurmountable technical banners to 
ihe use of UI wage record data as an evaluation tool.... The UI system can provide extremely^accurate 
and reliable information on the long-term labor force experiences of program participants. - 

The two primary issues in using UI data to determine employment outcomes are: how out-of-state 
employment can be addressed; and the lack of coverage of certain types of employers or en^Pl^y"*^"^; 
including federal civilian and military employees. U.S. postal service workers, railroad employees, most 
independent contractors, employees of religious organizations, and self-employed individuals. 

Tlie Florida Employment Training Placement Information Program (FETPIP) provides a model for 
using UI wage records to monitor employment outcomes of postsecondnry completers. (Flonda uses 
this system to determine employment outcomes of leavers as well.) 

To supplement the UI data and to adjust for the incomplete coverage, the state has i^'J^'S^'^ 
cooperative agreements with other state and federal agencies to extend the coverage ^.^ /ETPIP to 
employers nof included in employer records. Additionally, employers of students identified through 
the wage records are asked for two data elements on each employee-specific occupation and county. 
Finally, institutions survey graduates/completers for whom no wage data are found m an attenipt to 
account for completers who are unemployed, self-employed, not in the labor market, or employed out- 
of-stnte. 

The advantages of using UI wage data in Florida derive from three sources: com, data quality, and 
burden reduction. FETPIP is much less costly than earlier systems that attempted to collect analogous 
information. Typical costs for collecting training-related outcome information for just yocationa 
program completers and leavers averaged $17 per student, and the response rate was typically around 
20 percem FETPIP co.sts less than $3 per student and routinely traces close to 90 percent of the 
completers and leavers. In the past, the state budgeted $6 million annually to document comphance 
with the state's vocational placement standard, while the FETPIP budget is around $300,000 per 



year.-''' 



J. Trou. C.l-,, arul Stevcm. D. A Feasibility Study of the U« of Unemn lovment Insurance Waee-Record t)nca -is m Evaluation Tool for JFTA. 
National Curamission for Umploymem Policy. Research Report Number «H)-OZ J.tnutry IWl. 
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with ins.i,u,ional ..uden. records. Jrning W ori^s "rte' eLmi^M H^' '''"'t ""'^"^ 
characteristics and pos.secondary experiences. ^ ''^ ™"'P'^'"/8''«iume 

.n. Chive were Le, ^d. .He i ^tr^i^^^^^^^^^^^^ 

cL of maintaining ^h^'^^JlT " 1 eT^^^^^^^^^ ^ttf S ■^^''"S "1 ""''"'"'^ 
experience indicates the cost might be even less ' "=P""'='' 

is consistentlith the evd on aiiS^^^^^^^^^^ 

worlcing in an appropriate field, i mly blTsi^iLfllu^^ t "'u^-'" " ""'^ *^ 

with a lower level of training ^ "'"h""' ''"ining. or 

iliXLVauZ^^^^^^^^^^^^ -f"' "„g comparable statist., across 

system. TTis is an eroZ~em7n a^^^^^^^^^^ -'•^'-^ 
The inclusion of for-profit Doslsecondnrv 1^?! r , " "i"'- 

FETPIP.« ObviousW « of hrm",h^„ " P^"""'"' i-nplemented bv 

several years ^' -""hodology on a nationwide basis could not be achieved Ib'r 
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2. Transfers to Another Institution 



Section III notes that tracking transfers from one institution to another seems to be very problematic 
since only 15 percent of institutions reported having this capability (table 1II-4). As m job placement, 
the primary method currently available to individual institutions for obtainmg statistics on further 
postsecondary enrollment is a periodically administered cross-sectional fo"ow-"Pj;,"iy of former 
students Such studies are subject to the difficulties noted earlier. Additionally. NCES survey ot 
institutional reporting capability suggests that far fewer institutions use follow-up surveys to ascertain 
transfer rates than use them to determine employment outcomes. 

Alternatively, states that have statewide student record systems can automatically determine transfers 
from any institution included in the statewide system to any other institution in the system. 1 his, 
therefore, is a rational approach to the student transfer issue. For most current statewide systems, 
however only public institutions are included. As a result, there is a large information void concerning 
transfer^'to and from private postsecondary institutions. Private institutions would be put mto an untair 
situation if transfers to another institution were considered a completion for determining an institutional 
graduation/completion rate. Obviously, this particular void could be filled if states included all in-state 
postsecondary institutions (public and private) in their statewide systems. This would account for a 
significant proportion of all transfers, and, by developing data-sharing agreements with neighboring 
states many of the remaining transfers probably could be accounted tor. 

Another emerging approach is the "electronic transcript" now being developed by several college and 
university consortia under the aegis of the American Association of Collegiate Registrars and 
Admissions Officers (AACRAO). Under this method, machine-readable performance and enrollment 
Information about individual students can be transmitted from one institution to another to permit the 
generation of aggregated transfer statistics.^ However, the use of this system for this purpose has 
not yet been demonstrated. 

3. Pa.ssage Rates on Licen sure Examinations 

Section III demonstrates many of the problems in collecting information on results of licensure 
examinations through interviews conducted with a limited number of licensure boards. These problems 
include state laws on confidentiality and the vastly different policies regarding reporting of results ot 
examinations administered by state licensure boards, professional societies, and testing companies. 
There is little question that institution-level passage rates on licensure or certitlcation examinations are 
relevant indicators of studem achievement, and, when administered at a national level, would result in 
comparable inter-institutional statistics. It must also be noted, however, that they tend to be 
concentrated in health, education-related, and engineering-related occupations although they also are 
required by some other occupational and professional fields. Even within fields, it may be that only 
a small percentage of practitioners are typically certified or licensed. For example, civil engineers must 
be licensed to practice, but electronic engineers do not need licensure. Thus, coverage is a problem. 



^ask Force on the StandarfiiBHoo of Posuecoodaty BiiKstioo Bectronic Dsns Ewhange. A Guide to the Implementation of the AACRAO Bf ctronic 
Transcript. American Association of Collegiate Registrare and Admission* Ofticei^ 1990. 
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In a small number of programs, principally education and social service occupations tlie award of a 
degree from an officially certified progmm itself constitutes licensure. 

To summarize, then, the major problems associated with the feasibility of reporting licensure pass rates 

^LUr'T"'"^' """'^ """y- obtaining inslitmionaU^re^eTof 

extri „L Tk"" r" "'='»">'«'°"^- R^g^'ding 'he first, no. much can be S Pas 
expenence w,th baccalaureate programs Qirincipally in Tennessee, where such statistics are reouired 
mdicates that approximately one-third of all major fields of study in a typical univet^ w Zv hi 

S^t"h I'j'.r"""' " "SS^'^S'''^ '^'^^ by working individually 

^elTee V^^TZnT'^'t'' administering certification or licensure examinain; n 

aJSes robtl' thfr"' "TT^""'''" *i>h ""ny such associations 

ana agencies to obtam the required information. Establishing such relationships however takes 

considerable time and resources and is not always successful; some agencies win simply refu e o 

students to supply this information to the institution via a survey questionnaire This method , -,„ 
obtain partial information that might otherwise not be available at al?. but is .su^ec. i„ L s I L ,m 
problems of partial response and associated respon.se bias. sifenilicinl 



D. Assessment Outcomes 



Cunemly, no national instruments or methods are available to reliably report standard levels „f 
ach^vemen. for postsecondary students. Proposals for the development of s ch met ods are re n.lv 
. 'f" '"" '^"y 'h= development of rs^rateey" ^ 

Tl^:'7;^ 7^^°"' ■""•"-^'-S ^"""^ -ong gradu'ating col^^'i , ^ 
an assesment it is estimated, is more than 5 years away and is intended to sample eraduatina seniors 

soIvSu Ire ulld ihf ff '"f''""™^ '"'"'''"S. communication, and problem- 



theVmay ^otSc^^^^^^^ Zts ^'^""'"'"y '"'"'"""ns. for some institutions, 

i.„.ir,/ • but inputs. Students emer postsecondary education with widelv dkmrnw 

levels of skills m reading, writing, and mathematics. Indeed, many postsecondrr; in^SonSc! 
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students primarily on the basis of their high skill levels and general learned abilities. As a result, 
conventional, normative, standardized testing of a student's basic college skills m postsecondary 
institutions would not be very meaningful as a student outcome. 

One alternative to conventional normative or criterion-referenced testing might be to take a "value- 
added" approach to the measuremem of college-level skills. This approach would measure entermg 
students' skill levels on a standard assessmem instrument that has been designed to have a very low 
tloor and a very high ceiling. Retesting students after a predetermined period of attendance (defined, 
perhap.s in terms of number of credit or contact hours attained) would indicate changes m students 
scores and this change would be considered the student outcome. Of course, for this approach to be 
valid, change .scores would have to be scaled for equivalency from each successive achievement level, 
a scaling technique that might require con.siderable development. 

Another possibility is to develop standard college-level minimal competency tests for each of the 50 
program areas listed in the Clas.sitication of Instructional Programs, or, if necessary, lor .specific major 
fields of study. Use of such tests could indicate the percent of students who achieved a minimum 
knowledge base in their field. Thus, it would be tantamount to having llcensure/eertitication 
examinations that covered all fields and were appropriate, but different for all institutional sectors. 

In addition to the significant and costly research and developmental work that would be necessary to 
effect either of these asses.sment programs, assessing students on a nationwide basis to obtam institution 
scores would necessitate its acceptance by all segments of the postsecondary community. Given the 
current climate of postsecondary education and the widespread interest In assessment, however, these 
may not be entirely intractable problems. 
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SUMMARY AND CONCLUSIONS 

A. Summary 

employment m a par.icular vocation, tmde, or professional rSd" ' 

such a program obtn.n employment in tite occupation for which they are tr^l: and 
(E) other inslilulional outcomes that may be appropriate." 

i!^d!ca~;heret:ht^^^^^^^^^^ — ^ P~nd„ ry education 

Institutional accountahnit/rnd consume Z 'V "T""^ " P'=™'" "f 'i™--'- 

of data. How .state" use student ^^ZZr , "'™'"'"' '"'"'"'y '"^ •W'' Ivpe 

to institutions wet Z ttcriled ' """--"""^ 

So«i7i.wL%ldrtr """"""^ institution-leve, student outcome data collection and 
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. it is possible for most institutions that disclose or report graduation/completion rates to 
report them by a student's personal characteristics (e.g., gender, race/ethnicity, and age) and 
enrollment characteristics (e.g., field of study, especially initial major field of study, 
attendance status, and residency status). Fewer than half the schools could report these 
rates by students' financial aid status or school or academic division; 

- almost 80 percent of all institutions reported being able to determine employment outcomes 
for graduates/completers; 

- 58 percent of all institutions reported being able to determine if graduates/completers 
obtained necessary or appropriate licenses or certifications; 

. of the institutions that could report graduation/completion rates, only 15 percent said they 
could determine if a student actually transferred to another institution. Thus, ot :ill the 
student outcomes specified in section 103 of the SRK Act, transfers to another institution 
seemed to be the most problematic for most institutions; 

- 72 percent of all higher education institutions that are non-specialized and serve 
undergraduate students assess the basic college-level skills of their students; 

- 39 state higher education agencies (including Puerto Rico) either have or are developing 
the capability to report graduation rates for post-secondary students. Of these, 19 regularly 
report graduation rates, 6 report them sporadically, and U will report them within the next 
few years; 

- 46 communitv collece and public vocational/technical school oversight organizations 
reported collecting graduation/completion data. These 46 state agencies represent about 90 
percent of all public postsecondary institutions with vocational programs in the 50 states 
and the District of Columbia. Thirty-four of these 46 organizations reported actually 
computing a graduation/completion rate; 

- 33 state agencies reported collecting placement data for public postsecondary school 
vocational completers. Twenty-seven of these agencies, representing 46 percent ot public 
2-year and less-than-2-year schools, collect data on training-related placement; and 

12 state agencies reported collecting licensure information. 

The quality of curremly available student outcome data was evaluated in terms of their utility for the 
consumer protection purposes implied in the Student-Right-to-Know Act and for mstitutional 
accountability uses. The criteria applied in the evaluation included the data's validity or relevance, their 
consistency, and their representativeness, where possible. Currently available institution-level student 
outcome data also were evaluated in terms of their apparent inter-institutional comparability and their 
completeness of coverage. 
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Several problems that adversely affected the quality of institution-level student outcome data were 
identified: 

■ "se of surveys of graduates to assess post completion outcomes (training-relaled 
employment, earnings, licensure) are generally unreliable and may be biased since response 
rates to these surveys are fairly low and there is little inforaiation on the characteristics of 
non-respondents; 

- there is a great deal of definitional inconsistency among institutions and institutional sectors 
nationally and within states. This seems to be the case for almost all student outcome 
statistics; 

- although the ACE survey indicates that 71 percent of the institutions that were required to 
report assessment results to a state oversight agency felt their tests were comparable to 
those of other institutions, this represents only 18 percent of all postsecondary institutions 
Furthermore, there were no indications of the extent of the comparability (e.g., only peer 
mstitutions within the state; all other public institutions in the state; cross-state peer 
institutions). Additionally, since this survey was directed only to higher education 
in.stitutions, the results may not be relevant to other types of postsecondary institutions; and 

- problems of reporting licensure examination pass rates were severe. They included state 
laws on confidentiality and the vastly different policies regarding reporting of examination 
results of state licensure boards, professional societies, and testing companies that 
admmister licensing examinations. Additionally, approximately one-half of professional 
fields and virtually all traditional academic fields of study would be excluded from such 
reporting since licensure examinations are not required to work in a related field. 



B. Conclusions 



From the literature in the field, earlier national postsecondary legislation, reported stale uses of siud.nt 
performance information, and the potential for institutional use of these data there is little doubt that 
these data are or could be uselul and meaningful to many and varied interest groups in po.sisecondary 



The very fact that so many institutions and states are engaged in collecting and reporting student 
outcome information is an additional, powerful indicator of the perceived need for, and desirability of 
student performance information ^ 



The graduation/completion rates and possibly the student assessment data currently collected by 
insmutions and states may be sufficiently valid, consistent, and unbiased to be useful to institutions or 
states for self-assessment or oversight purposes. However, it was not possible, within the scope of this 
tlT T ^'"f"'.'^'' perspective. Training-related employment data, licensure pass 

leavers) are not useful for any purpose, given the problems and costs of obtaining cons stent and 
demonstrably unbiased data through surveys of completers. ""mmcci anu 
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In geneniU it may be concluded that the use of any of the currently available data for consumer 
protection purposes is highly doubtful. In order for students or other consumers to make an mlormcd 
choice among poslsecondary institutions, the data on which these choices rest must be comparable. 
From repims of institutions and states, the procedures used to collect student outcome statistics vary 
considerably across institutions and there is a substantial degree of definitional mconsistency as well. 

The Student-Right-to-Know Act, however, does not require disclosure of outcome statistics until July 
1 W. There is si^me time, therefore, for bringing about changes in delinitions and methodologies that 
will help create a more comparable and complete set of institution-level student outcome statistics at 
a national level. 

To establish a national set of postsecondary student outcome statistics, representatives of broad 
seuments of the postsecondary education community advocate establishing definitional consistency and 
standardized data collection procedures and methodologies at a national level that could be 
implemented at the institutional and/or state level. 

1. Specific Suggestions 

Suugestions for specific deimitions and approaches to data collection procedures and methodologies 
tha*t could introduce quality, comparability, and maximum utility imo the nationwide reporting ol 
iastitution-level student outcome statistics include the following: 

Graduation/completion rates 

(1) Utilization of a forward-looking cohort methodology for determining a valid and 
meaningful in.stitutional graduation/completion rate. This methodology would enable 
institutions to report the percent of a cohort completing, the percent persisting from one 
year to the next, and the time to completion. 

(2) Establish separate but parallel cohorts of full- and part-time degree-.seeking, entering 
undergraduates. The con.sideration of part-time students is important because almost 30 
percent of students who enroll for the first time in any po.stsecondary institution, enroll 
part-time. In some institutions this percentage is much higher. Furthermore, 62 percent 
of institutions currently include part-time students in their graduation/completion rates 
and attendance status can change even within a single attendance period. While it is 
recognized that the Student-Right-to-Know Act specifies determining 
graduation/completion rates only for students who enroll initially as full-time students, 
this is inconsistent with current trends in postsecondary education. 

(?) Report or disclose all graduation rates by meaningful, static, student characteristics such 
as gender, race/ethnicity, age, and intended major field of study. These characteristics 
can easily be identified when the cohort is established. 
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(4) Reporting graduation/completion rates by actual major field of study, academic division 
or any variable that could change frequently over a short period of time is particularly 
difficult using the suggested forward looking cohort methodology. Therefore, reporting 
by these variables is not suggested. 

(5) Establish separate cohorts for individuals who enroll for the first time in nn^ 
postsecondary institution and individuals who are first time in the institution, that is, 
transfers into the institution. Report graduation rates separately for each cohort. 

(6) Establish cohorts at a single point in time in the school year, rather than at multiple 
points. 



Employme nt rates of oraduates/comntcters am ' t ;irninus 

An efficient and cost-effective methodology for oecermining employmem outcomes of 
graduates/completers is through electronically linking files containing 
graduates/completer records for the state with available Unemployment Insurance (Ul) 
Wage>Record Data. Surveys of individuals after they leave the institution, althouuh the 
prevalent methodology for determining employmem outcomes of completers, does not 
generally produce consistent and unbiased data due to low response rates. 



Transfers to another institution 

(1) The transfer of a studem to another, perhaps higher level, institution should be a 
separate student outcome. The percentage of students that transfer should be reported 
separately from graduation/completion rates. 

(2) Statewide student record systems that can automatically determine transfer from anv 
institution included in the statewide system to any other institution in the system is a 
rational approach to this issue. To make this a viable and complete system, all in-state 
postsecondary institutions (public and private) would have to be included in a statewide 
system. This would account for about 75 percent of all transfers. 

(3) Electronic transcripts through which machine-readable performance and enrollment 
mformation about individual students can be transmitted from one institution to another 
IS another developing method for tracking transfers. Utilization of these transcripts by 
the initial enrolling institution would permit the generation of aggregated transfer 
statistics. However, this system has not been tested for this use. 
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Pass rates on licensure or certification examinations 



There is little question that institution-level pass rates on licensure or certification 
examinations are relevant indicators of student achievement and, when administered at 
a national level, would result in comparable inter-institutional statistics. However, the 
uneven coverage of licensure/certification requirements across fields and the difficulties 
in negotiating institution-level pass rates with the multiplicity of licensing and 
certification boards are currently prohibitive obstacles, particularly for schools with 
multiple programs. Additionally, many licensure/certification examinations arc 
administered at the state level with each state having its own examination and criteria 
for passing. Thus, even were the data collectable, their comparability would be 
problematic. 

Assessment of student learning 

Consider the development of a post.secondary assessment program. It could entail a test 
of basic postsecondary-level skills that could measure changes in skill levels of students 
regardless of their initial skill level upon entrance into the postsecondary institution or, 
alternatively, multiple tests of subject-matter achievement. Either of these approaches 
would be a long-term and costly effort, but one may be tenable in view of the current 
interest in postsecondary assessment. 



2. General Suggestions 

For graduation/completion rates and other institution-level .student outcomes to be as meaningful as 
possible to all potential u.sers, two additional suggestions are warranted. First, student outcomes are 
related not only to the efforts and effectiveness of the postsecondary institution, but also to the 
characteristics of the students in the institution. In fact, much of the research on identifying correlates 
of student outcomes has shown that student characteristics are better predictors of student outcomes 
than institutional characteristics or processes. Thu.s, it is vital that contextual information describing 
an institution's student population be considered in conjunction with its student outcome data. Indeed, 
it may be that for truly valid comparisons among postsecondary institutions, institutional outcome 
statistics should be adjusted to account for differences in characteristics among the students served by 
the in.stitution. 

Second, this report revolved around the need for comparable data among postsecondary institutions for 
the consumer protection purposes underlying the Studem-Right-to-Know Act. Yet the Act itself 
promulgates differential collection and reporting of institution-level student outcome statistics among 
postsecondary in.stitutions. In particular, the distinctions made by sections 103 and 104 of Title 1 of 
the Act on the basis of an institution's athletically-related student aid policies should be reconsidered. 
If information, such as graduation rates by gender or race/ethnicity or average graduation rates are 
useful, they should be available to all potential students and their families and for all postsecondary 
institutions, not ju.st to potential student athletes for iastitutions providing athletically-related student 
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aid Addilionally, they should be available on a national basis (as those required in section 104") 



7f' '™™""8 suggestions made in this report for all postsecondary institutions in the 
™1 can ietl.nLltir^^^^^^ As a resuMhere is little hope'tha. even graduatioSr^^ct o 
i^es can be ful y established for reporting in 1993, as specified in the Student-Right-to-Know Act 
TTte most reahst.c expectation is that institutions can immediately begin tracking entering studenBnnd 

tTuW LTZ^eSn^l ?^ 4.year'instiSns e !„" 

tec™se ^lZr. H ^ °" """"" P"""' °f """"'l This is possible 

because persistence and graduation rates are intra-institutional parameters that can be validly determined 
by each postsecondary institution acting independently. ^ determined 

may'tVe'more"t'r' of establishing them mitionwide will be more difficult and 

may taKe more time. Each of them requires some developmental effort and involves a consider«hle 
degree of cooperation between postsecondary institutions, state postsecond^y agenc es " n^^^ 
federal agencies. Most are possible to collect and report, however, using apZp far,^e^^^ 
.ns those that were suggested here. Given the importance of these st tis.ics anZe n ed to tve S 
nvailable to the nation, the proce.ss of establishing them should be started 
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APPENDIX A 



Summary of the Definitions Worldng Group Meeting 
for Sections 103 and 104 of Title I 
of the Student-Right-to-Know and 
Campus Security Act 
March 28, 1991 



The first meeting of the Definitions Working Group for sections 103 and 104 of Title I of the 
Student-Right-to-Know and Campus Security Act (P.L. 101-542) was held on March 28, 1991. 

A set of materials, sent to invitees prior to the meeting, identified the agenda and raised some 
general questions concerning the direction the definitions should take, several definitional issues 
that had been identified, and possible alternatives for the various terms that were to be defined. 
Additionally, invitees were asked to consider "safe harbors" - that is, guidance on those actions 
that postsecondary institutions could take to be in compliance with PL 101-542 prior to the 
issuance of formal regulations by the Education Department (ED). 

This summary follows the sequence of topics discussed. It first describes the group's 
consideration of the general questions, details the discussion of specific definitions, and, finally, 
indicates the group's suggestions regarding "safe harbors". 

General Directions/Overarching Concerns 

I- The first question was whether comparable definitions of graduation/completion rates, and 
methodologies for determining them, should be required of all institutions regardless of 
their type. While there was recognition of the different missions of different types of 
postsecondary institutions such as 4-year institutions and 2-year community colleges, there 
was a fundamental agreement that PL 101-542 is a consumer protection law. There was. 
therefore, a strong consensus that definitions and procedures needed to establish 
comparable dat a should apply to all institutions. 

Additional discussion concerned short programs, those of a 9-months or less in length. 
Some schools may offer both short and longer programs while others, such as private 
career schools may have only short programs. Since students are oftentimes admitted to 
these short programs continually over a period of time, especially in the private career 
schools, there was concern that the methodology for determining graduation/completion 
rates for these programs should be kept as simple as possible while still ensuring that 
accurate, representative, and even comparable data are obtained. Thus, there was 
agreement that some accommodation for determining completion rates be offered to all 
schools that operate short programs. 
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n. The second general question was whether interim and/or concomitant statistics for 
graduation/completion rates should be identified and defined. The need for an interim or 
preliminary statistic, until schools have sufficient time to begin collection and follow a 
cohort through to completion, was fully recognized. However, the group also clearly 
stated that an interim statistic should not serve as a substitute for eventual determination 
and reporting of graduation/completion rates. Identifying concomitant statistics that could 
be reported in addition to graduation/completion rates was supprorted, but it was also 
believed that reporting of these additional statistics should be the prerogative of the 
reporting institution and not mandated by federal law. 

The one interim statistic identified was the re-enrollment or persistence rate of the cohort 
of full-time, entering students for each year the cohort is followed. Re-enrollment or 
persistence rates were also considered to be useful concomitant statistics, even when 
graduation or completion rates become available for a given cohort. 

There was also a brief discussion of possible contextual parameters that institutions might 
report, that could lead to the development of contextual indicators (composites of several 
institutionally specific variables). Again, however, these were perceived by the group as 
something an institution could report along with their graduation/completion rates, to help 
explain their graduation/completion rates and to facilitate comparisons of rates among 
similar institutions. But, institutions should not be required to report these to meet the 
intent of PL 101-542. 

III. The third general question concerned the need to accommodate definitions and procedures 
to different length programs. In considering this question, the primary suggestion was 
targeted toward schools with only short programs (9 months or less in length) and roiling 
enrollments (starting a new class every few weeks). One suggestion for dealing with this 
situation was that in schools with only short programs and rolling enrollments, all .students 
enrolling during the first quarter of the year (July through September) would be used as 
the cohort. At the end of the year (June of the following year) the number of students 
in the cohort who completed within 150 percent of the normal time for completing their 
program would be determined and this would be the number of 'completers' in a 
graduation rate calculation for that reporting year. Schools enrolling students on the 
quarter or semester system would use the same cohort as all other schools and colkges. 

IV. When asked whether there was any methodology other than the use of a cohort for 
determining valid graduation/completion rates, the consensus was that a forward-looking 
cohort methodology was the only appropriate method for determining an institution's 
graduation/completion rate on a basis comparable with other institutions. This consensus 
was reached after considerable discussion of a possible backward-looking approach. Tlie 
backward-looking approach proposed by one participant would assess, at the end of each 
year, the proportion of all students enrolK i during the year for whom the ratio of credits 
attempted to credits earned was 150% of the number of credits needed for obtaining a 
degree or other formal award. 
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Definitions/Procedures Necessai^ for Defining a Cohort 



The general agreement of the group for developing definitions was that, wherever possible, 
definitions should be consistent with those already in use by institutions either because of IPEDS 
reporting requirements or because of extant ED regulations for federal student assistance 
programs. Further, it was agreed that appropriate IPEDS definitions should generally take 
?Xc^oT ""'^ ''"'^^"^ "^""^ collection would facilitate additional reporting and 

Entering Student 
Definition 

by thTgmu^ of definitions of an 'entering student', three alternatives were seriously discussed 

1. One definition was the IPEDS definition of 'first-time' student. This definition 
mcludes only students who enroll for the first time in any postsecondary 
institution. ^ 

2. A second definition suggested would, in addition to including the IPEDS 'first- 
time* student, also include any student who transferred into the institution from 
another school, but with fewer than 15 transfer-credit hours. 

3. A third possibility would include any student who enrolled in the institution for 
the first-time with no credits transferred into the enrolling institution, regardless 
of his/her previous postsecondary status. (Under this definition, a student with a 
baccalaureate degree enrolling in a career school or community college would be 
included m the cohort. This could also happen under the IPEDS definition of 
tirst-time , if the institution was unaware of the student's previous postsecondary 

status. However, institutions should, and generally do, ask students about any 
previous postsecondary enrollment.) 

The group recommended the adoption of the IPFDS definition of 'first-time^ student. 

There were three reasons for this recommendation. First, institutions, especially higher education 
^T'r^ are well-equipped to identify students who meet the IPEDS definiLn^of a flm-t me 
smdent since they have been reporting enrollments using this definition for a number XeaTs 
Second y identifying and including students who transfer in fewer than 15 crX wild be 
difficult for institutions to do. Moreover, doing this could conceivably result in inconZencies 
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Furthermore, the group believed that this definition encompassed those students for whom 
Congress believed consumer information was particularly important and the graduation rates of 
these students reflects the institutional "outcomes" measures Congress intended. 

Procedures 

1, What 'first-time' students should be included? 

In discussing what entering student should be included, it was the consensus of the group that 
students enrolling in the fall onlv be included. This would be consistent with the IPEDS fall 
enrollment survey. Also, in concordance with IPEDS, first-time students who enrolled in the 
previous summer would be included with fall first-time enrollees. 

2. When, i.e., after what period of time in attendance at the institution, should a 'first-time' 
student be included in a cohort? 

This question was not addressed by the group. 

Full-time Student 

Definition 

There was little formal discussion of the definition of Mull-lime'. However, the general approacli 
of being consistent with IPEDS and ED regulations, suggests that students enrolled for 12 or 
more semester credits or their equivalency in quarter or contact hours be con sidered 'full-time'. 

Procedures 

Tliere was a brief discussion of procedures for identifying full-time students and their 
maintenance in the cohort. As a result of this discussion: 

The group recommended that students who enroll for the Tirst-time' as full time 
students in the fall remain in the cohort regardless of the students' subsequent 
enrollment status. 

This corresponds with Congressional staff's statements of Congressional purpose. 
Degree/Certificate Seeking Students 
Definition 

As with the concept of 'full-time', there was little formal discussion of a definition for 
'degree/certificate seeking' students. If the general approach of being consistent with IPEDS was 
applied, however, any .student enrolled in courses for credit who is recognized by the in stitution 
as seeking a degree, certificate or other formal award would be included. 
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Completer 



Definition 



Initially in the discussion of the problems with short programs, it was agreed that, in schools with 
a mix of short and longer programs: 

a student who receives any degree or other formal award offered by the enrolling 
institution would be included as a completer regardless of the degree the student was 
seeking upon entrance to the histitution. 

There was a lengthy discussion of who else, in addition to degree recipients, should be 
considered a completer'. This discussion focussed primarily on the provision of Ihe Student- 
R^ht- o-Know Act that indicates that 'completers' include students who transfer to another 
eligible institution for which the enrolling institution has provided 'substantial preparation'. 

The discussion of whether to recommend inclusion of transfers as part of the definition of 
completers recognized the extreme difficulties institutions would have in determining a student's 
ransfer status once the student has left the enrolling postsecondary institution. It was noted that 
institutions are usually unaware that a student has transferred. There was a stronger sense 
^Z^^u .^^''^'"f "8 le^^t ^^^nsfcn. to a higher level postsecondary institution,' 

would be detrimental to many community and junior colleges and did not meet either the spirit 
or the wording of the law. *^ 

routl11?m'n''f" '""^^'"^ suggestions as to how institutions 

could determine former students' transfer status and what kind of transfer could be considered 
a positive institutional oujcome. Two kinds of transfers were considered by the group. 

1. Transfer from the enrolling institution to any other postsecondary institution 
regardless of the receiving institution's level. 

2. Transfer from the enrolling institution to another postsecondary institution which 
IS at a higher level than the enrolling institution. 

^ri nf IT'T 'T''"^'' ^"'^ ' ^'''^ postsecondary institution as 

part of the dei.nition of completers. 

The final recommendation of the group regarding the definition of a completer was: 

Any student who receives any degree or other formal award from the enrolllne 

I T ""^ ^''^^ ^"^f^*-* fr^*" «he enrolling institution 

to a higher level postsecondary institution. 
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An additional recommendation of the group was that: 

In reporting graduation/completion rates, schools should report rates separately for 
students who remained at the institution and received a degree or other formal 
award and students who transfer to a higher level Institution. 

It was noted that these rates should be additive when determining an individual institution's total 
completion rate since they would have a common base. 

Raised, but left unresolved by the group, was the question of a possible mechanism through 
which an institution could learn of transfers to another institution. Possible mechanisms 
identified v/ere imposing, by law or regulation, the requirement that a transfer institution report 
a transfer to the enrolling institution or encouraging voluntary agreements among institutions for 
the exchange of this information. 

Normal Time of Completion 

Definition 

Arriving at recommendations for a definition of 'normal time of completion' engendered a great 
deal of conversation and several possibilities. While the prevailing sentiment was that a fixed 
time for completion should be recomn.v-nded, fixing a specific 'normal lime' was a difficult 
process. 

The difficulty was identified primarily for baccalaureate degree programs and emanated from two 
sources. First, while a baccalaureate is generally perceived as a 4-year degree, data indicate that, 
on the average, a baccalaureate degree is actually taking students who consistently maintain full- 
time attendance status about 4.5 years to complete. This raises the question of whether 'normal' 
time should be the perceived (and usually advertised time) of 4-years, or it' it should reflect the 
actur.! time of 4.5 years? 

Secondly, many schools have baccalaureate degree programs that are explicitly 'advertised' as 
taking five years to complete, either because the necessary coursework requires this amount of 
time or because other programs, such as coop programs or internships, are available to students 
while working on their academic degree. Further, these 5-year baccalaureate programs usui lly 
coexist with traditional 4-year programs. 

Two suggestions were made to address these difficulties: 

1. Institutions could report completion rates of a cohort after different time periods 
have elapsed (4, 5, 6 [150% of 4 years], and 7 [150% of 4.5 years] years for a 
baccalaureate degree; 2, 3, and 4 years for an associate degree). 
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2. The 'normal lime for completion' could be based on the published length of 
programs in which the majority of students at the institution are enrolled. In 
conjunction with this, it was suggested that even if a small proportion of students 
were m shorter or longer programs at the institution, the 'normal time for 
completion' for the institution would be based on the program length of the 
majority of students. (For example, if the majority of students were in a 5-year 
engineering program at a given institution, 5 years would be the 'normal time of 
completion'.) 

The consensus was that a fixed period of time . for compabUity and simplicity - needed to 
be established for both baccalaureate and associate degree programs. 

While recommendations for fixing the 'normal ti m e of comnletinn " for a hnccalnnrPntP H.ar.. 
wavered between 4 and 4.5 vea^he group w.^ fair l y united in re^nmmending 2 venn. th. 
normal time of completion' for an nssociate d^or^^ Thus, 150% of normal time foTli 
baccalaureate degree would be either 6 or 7 years and 150% of the normal time of completion 
tor an associate degree would be 3 years. 

It was also suggested that institutions could report, if they chose, supplemental data of the 
graduation/completion rates for additional time periods of shorter and longer duration than thai 
established m the regulations implementing PL 101-542. 
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"Safe Harbors" 

The group also addressed the question of what guidance could be provided to postsecondary 
institutions immediately to help them meet the intent of PL 101-542. Participants were asked 
to suggest specific actions institutions could take to be in compliance with PL 101-542 prior to 
the issuance of formal regulations by the Education Department (ED). This resulted in the 
following recommendations: 

L Institutions should establish a cohort of first-tlme, full-time, degrec-seeking 
students as of July 1, 1991 (fall, 1991). 

2. Schools with only short programs of 9-months or less in length should 
determine and have available for reporting in 1993 graduation/completion 
rates for the 1991-1992 academic year (July 1, 1991-June 30, 1992). 

3. Schools whose predominant programs are a year in length should determine 
and have available for reporting graduation/completion rates of the 1991 
cohort by July 1, 1993. 

4. Sch'^ols whose predominant programs are two years in length should 
dcf nine and have available for reporting graduation/completion rates of the 
1991 cohort by July 1, 1994. 

5. Schools whose predominant programs arc four years in length should 
determine and have available for reporting graduation/completion rates of the 
1991 cohort by July 1, 1997 or 1998. 

6. Schools that cannot report graduation/completion rates by the first reporting 
year specified in PL 101-542 (1993) should have available for reporting in 
1993 the percent of students in the 1991 cohort who are still enrolled in the 
spring of 1993 (the re-enrollment or persistence rate of the 1991 cohort). It 
will do the same for each succeeding cohort until graduation/completion rates 
can be fully reported. 

7. Schools that have the capability of reporting graduation/completion rates for 
their first-time, full-time degree-seeking students for an appropriate cohort 
in 1993 (a 1990 cohort for associate degree recipients and a 1986 or 1987 
cohort for baccalaureate degree recipients) should determine these rates and 
have them available for reporting by July 1, 1993. 
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Summary of the Public Meeting on DeHnitions 
for Sections 103 and 104 of Title I 
of the Student-Rlght-to-Know and 
Campus Security Act 
May 14, 1991 



A public meeting on definitions for sections 103 and 104 of Title I of the Studenl-Right-to-Know 
It^ """k M "T"' ^"^-^ ""-^"^^ °" ^''y l"'!- 'he second of two 

was TeW on M*r* T wT.'''''' P"'"'™"'"'^ immunity. The fitsi nteeting 

A set of materials, sent to invitees prior to this meeting, included a summary of the firet mectine 

.°[-'Tu ■J"""'"" P»rt"=iP»ts in the first meeting. n,e new 

m tertals tdemtfied U.e first group's resolution of four general questions concerning the direction 

fi.^. f ur A, , h'h T";"' ''^'■'"""'"^ recomt^ended by the 

first group. Also included was the first group's suggestions regarding "safe haibors" - that is 

guidance on those actions that poslsecondary Institutions could take to be in compliance with ?£ 

lUl-542 prior to the issuance of formal regulations by the Education Department (ED). 

Th,-. non-ED members of the first working group hod been limited to a few Individuals with 

SZr„d r'">? ""^'^ P— """^ ""-'ion "Ota. As a result, Tb y 

had first-hand familiarity with institutional and state reporting capabilities and practices The 
su^estions and recommendations of this group reflect this daU, orientation. TT,e second group 
differed somewhat from the first group in that: (1) there was wider represemation of "he 
postsecondary education community including Individuals who represenlea specific types of 
proprietary schools as well as the more traditional 2- and 4.year colleges; (2) there was more 
represemation of policy-level individuals. Including several state higher education recutive 
officers and association directors; and (3) two (NCAA and NAIA) intercollegiate athletic 
conferences were represented at this meeting. TTtese individuals were invited to participate n 
the general discus.s,on as well as to lead specific discussions of the waiver provision of section 

iilicirdTf[nilr'''''?h'''' r"""?' s*"""' ""^'""'"^ ^^^--^^^ 

specific defini ions. The primary focus, however, is to indicate how this second erouD's 
deliberations diverge from the recommendations and suggestions of the first working group 

General Directions 

I. '" 'I" first group, there was a strong consensus that standard definitions and procedures 
should apply to all mstltutions, regardless of their level or control in order to obtain 
comparable data. Both groups recognized the different missions of each of the differem 
types of postsecondary institutions, yet both expressed the sense that PL 101-542 is a 
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The second group also agreed with the first group*s belief that the methodology for 
determining graduation/completion rates for programs of 9 months or less in length should 
be kept as simple as possible while still ensuring that accurate, representative, and even 
comparable data are obtained. Thus, they agreed that some accommodation for 
determining completion rates be offered to all schools that operate short programs. 

The second general question of whether interim or preliminary statistics for compliance 
with graduation/completion rale requirements should be identified and defined was 
discussed at length by the second group. As with the first group, the need for an interim 
or preliminary statistic, until schools have sufficient time to begin collection and follow 
a cohort through to completion, was fully recognized. The second group also concurred 
that an interim statistic should not serve as a substitute for eventual determination and 
reporting of valid graduation/completion rates. The one interim statistic identified by the 
first group was the re-enrollment, retention, or persistence rate of the first cohort of full- 
time, entering students for each year the cohort is followed. This was felt to be a 
reasonable and meaningful statistic among the second group participants. 

Because the Act specifies that institutions must make graduation/completion rates 
available by July 1, 1993, the need for identifying a proxy graduation/completion rate (in 
addition to re-enrollment rates) was expressed by ED participants. They proposed using 
the ratio of the number of graduates/completers to the number of Hrst-lime entering 
freshmen 6-years earlier (for a 4-year program). This proposal was almost unanimously 
rejected on the grounds that: it was a completely inaccurate and misleading statistic and 
could be orders of magnitude different from the "true" graouation/completion rate. This 
could happen because, for example, the number of upper level transfers into the institution 
who graduate or complete would be included in the numerator (number of graduates), but 
not necessarily in the denominator based on a single cohort of entering freshmen. As part 
of this discussion, some of the more policy oriented participants suggested that, rather 
than reporting spurious and erroneous graduation/completion rates, institutions might be 
willing to assume the burden of searching through records of individual students to 
determine who was in the entering freshman class six years prior to 1993 (for a 4-yenr 
program), and, of these, what proportion graduated. However, the sense of the group was 
that this could be extremely burdensome, and was probably not worth the effort. 
Alternatively, the group recommended that institutions that were unable to report actual 
graduation/completion rates on the schedule specified in the legislation provide a "best 
estimate" of their graduation/completion rates until such time as they could report them 
for a defined cohort. Institutions that have been tracking students for a sufficient length 
of time to report actual graduation/completion rates should report these rates as 
required.Re-enrollment, retention or persistence rates were also considered to be useful 
and informative statistics, even when actual graduation or completion rates become 
available for a given cohort. 

With regard to the third general question concerning the need to accommodate definitions 
and procedures to different length programs, the first group had suggested that in schools 
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with only short programs (9 months in length or less) and continuous enrollments, all 
students enrolling during the first quarter of the year (July through September) be used 
as the cohort; at the end of the next year (June of the following year) the number of 
students in the cohort who completed within 150 percent of the nonnal time for 
completing their program would be the number of 'completers* in a graduation rate 
calculation for that reporting year. For example, the completion/graduation rate of 
sti^ents who enrolled in July through September of 1991 would be reported in July of 

While this suggestion was basically accepted by the second group, it was acknowledged 
that this methodology would result in only a partial picture of graduation rates in these 
types of schools, although these rates would be comparable (as much as possible) with 
those reported by schools with longer programs. 

IV. The consensus of the first group that a forward-looking cohort methodology was the only 
appropriate method for detennining an institution's graduation/completion rate on a basis 
comparable with other Institutions was implicitly accepted by participants in this second 
group. There was little discussion of this issue. 

Definitions/Procedures Necessary for Defining a Cohort 

In discussing specific definitions, the process used in this second group meeting was to explain 
the recommendations of the first group and solicit comments. 

r^lhlf dfr"'r ' '^TTf.'^ developing definitions had been that, wherever 
TiPi^r;^ '^''"''^ consistent with those already in use by institutions either because 

of IPEDS reporting requirements or because of extant ED regulations for federal student 
assistance programs, this approach was not readily accepted by this second, more policy-oriented 
group. As a result the degree of closure reached in the first group meeting was not achieved in 
this second meetmg. 

Entering Student 

Definition 

TTie liist group bad recommended the adoption of the IPEDS definition of 'first-time' student 
Using thtsdefimtion, only sttidents who enroll for the first time in any poslsecondary institution 
would be mcluded ,n a cohort for calculating graduation/completion mtes. Some participants b 
the second group felt that this definition did not reflect the situation in many 4-yearbsmulion" 
wtoe a substantial proportion of completers may have transferred into L institution 

freshl'n " r have no entering 

freshmen. While this problem was acknowledged by most of the group, the prevalem sentimeni 

c™r foI'nT"' '!,"'^"'' '"^ /^O-i^n/completio" 

calculation should not be recommended. Rath-.r was suggested that the graduation rates of 



A-II 



85 



upper level transfers could be reported separately at the discretion of the institution, perhaps by 
aggregating transfers into a single group and reporting the persistence/graduation/completion rates 
f^the entire group. Thus, implicitly this group concurred with using the IPEDS definition of 
first-time students. 



Procedures 

1. What 'first-time' students should be included? 



In discussing what emering studem should be included, the second group concurred with the 
consensus of the first group that students enrolling in the fall only be included. This would 
maintain consistency with the IPEDS fall enrollment survey. However, unlike the first group, 
this group felt that first-time students who enrolled in the previous summer should not be 
included with fall first-time enrollees. The basis for this sentiment was two-fold. First, in most 
higher education institutions the overwhelming majority of students enter for the first time m the 
fall of the school year. Second, students who enter for the first-time in the summer are 
oftentimes different from fall first-time enrollees. For example, they may be provisional students 
who would be admitted to the institution formally upon successful completion of the summer 



term. 



2. Wh n, i.e., after what period of time in attendance at the institution, should a 'first-time* 
.student be included in a cohort? 

This question was not addressed by the first group; however, during discussions of this issue in 
the second group, it was suggested by a community college representative that students should 
not be included in a cohort until they have been enrolled full-time for two comiguous terms, 
although this suggestion was not supported by the whole group. 



Full-time Student 



Definition 

As in the first group, there was little formal discussion of the definition of 'full-time' and using 
a definition consistent with IPEDS and ED regulations that students enrolled for 12 or more 
semester credits or their equivalency in quarter or contact hours be considered 'tuU-time' was 
accepted by both groups. 

Degree/Certificate Seeking Students 



Definition 

The discussion of a definition for 'degree/certificate seeking' students in this second group 
centered about the problem of unclassified students. In many institutions, relatively high 
proportions of first-time students are unclassified with respect to degree level. However, no 
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alternatives to the IPEDS definition of 'degree-seeking* that had been recommended by the first 
group were proposed in the discussion. Therefore, there was implicit acceptance of the IPEDS 
definition and 'degree-seeking' would encompass only students enrolled in courses for credit 
who are recognized by the institution as seeking a degree, certificate or other formal award. 

Completer 

Definition 

The first group had agreed that any student who receives any degree or other formal award 
offered by the enrolling institution would be included as a completer regardless of the degree the 
student was seeking upon entrance to the institution. In this second group meeting there was a 
great deal of discussion of this definition. Some participants expressed the possibility that use 
of the first group's definition could lead to a proliferation of degree programs at postsecondary 
institutions, particularly short degree programs. It wh- further argued that if this proliferation did 
occur it would be detrimental to the institution, the student, and even to trends in degrees 
awarded. Thus, it was suggested that to be considered a completer, a student would have had 
to complete the degree for which he/she was initially enrolled. Any degree/certificate received 
along the way would not count in the numerator of a completion rate. While no specific 
recommendations concerning this issue were formulated by the group, the group did seem to 
concur that using the process/definition recommended by the first group could indeed create a 
problematic situation. There was also some consensus that perhaps this problem could not be 
solved through definitions. 

As in the first group, there was some discussion during this second meeting of who else, in 
addition to degree recipients, should be considered a 'completer*. Again, this discussion focussed 
primarily on the provision of the Student-Right-to-Know Act that indicates that 'completers' 
include students who transfer to another 'eligible institution' for which the enrolling institution 
has provided 'substantial preparation'. 

Unlike the first meeting, there was no distinct resolution to the issue of transfers. On the other 
hand, there was no clear dissension with the final recommendation of the first group. That is, 
in reporting graduation/completion rates, schools should report rates separately for members of 
the cohort who remained at the institution and received a degree or other formal award and 
members of the cohort who transferred to a higher level institution. 

It had been noted by the first group that these rates should be additive when determining an 
individual institution's total completion rate since they would have a common base. 
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Normal Time of Completion 



Definition 

As In the first group, the prevailing sentiment in this second meeting was that a fixed time for 
completion should probably be specified. However, participants in this second meeting also 
agreed that because of the legitimate variations in 'normal time* for completion that are possible 
among institutions and among programs within an institution, institutions should report 
cumulative completion rates for a cohort over several years. For example, for a given cohort of 
baccalaureate degree-seekers, the proportion of the cohort graduating or completing after three, 
four, five, six, seven and even eight years should be reported. This recommendation was 
;»omewhat different from that of the first group. 

The first group had recommended that the 'normal time of completion" for a baccalaureate degree 
snould be fixed at between 4 and 4.5 years, and the 'normal time of completion' for an associate 
degree should be fixed at 2 years. The first group left reporting graduation/completion rates over 
several time periods to the discretion of the institution, suggesting that institutions could report, 
if they chose, supplemental data of the graduation/completion rates for additional time periods 
of shorter and longer duration than those that would be established in the regulations 
implementing PL 101-542. 

Additional Topics 

Discussion was held on several additional topics at this meeting: 

1. TTie specific requirements of section 104, particularly the provision that the Secretary of 
ED could grant waivers for institutional reporting under this section if the institution's 
graduation/completions were reported to an athletic conference that then reported them 
to the Secretary of ED and if these data were "substantially comparable" to data reported 
directly to the Secretary by the institution; 

2. The procedures that institutions should use in determining re-enrollment or persistence 
rates; 

3. Data that should be included in developing a student-unit record system for tracking a 
cohort of students; and 

4. A possible format for reporting graduation/completion rate data. 
1. Institutional Waivers 

The discussion of institutional waivers while centering primarily on the issue of 'substantial 
comparability' also elicited comments on the appropriateness of an athletic conference being 
designated as a data collection agent for the Secretary of Education. With regard to 'substantial 
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comparability there was a strong consensus that 'substantially comparable' would have to be 
Identical, both m definitions and procedures, to the data reported directly to the Secretary The 
appropriateness of designating an athletic conference as a data collection agent for the Secretary 
of Education was questioned by several of the participants especially those representing state 
agencies and institutions. While some participants noted that it might be appropriate for an 
athletic conference to report to the Secretary on athletes' graduation/completion rates, most 
participants agreed that it was not appropriate for an athletic conference to report the 
graduation/completion rates for the entire student body of the institution. As a result, there was 
a strong belief among participants that granting a waiver to institutions on the basis of their 
reporting to an athletic conference could result in an untenable situation. 

2. Determining re-enroilment or persistence rates 

It was recommended that renmrollment or persistence rates should be determined from the fall 
term of one academic year to the fall of the next academic year rather than from the fall term 
ot one academic year to the end of that academic year. 

3. Data elements for a student-unit record system that would meet the requirements 
of sections 103 and 104 

It was indicated that including the following data elements in a studem-unit record system would 
be sutficiem to meet the requirements of both sections 103 and 104 of PL 101.542- student 
name, identification number, race, gender, degree objective, attendance status (full- or part-time) 
at entry and subsequent enrollment periods, date of entry, academic program at entry, athletic 
scholarship status, sport, each term (period) of enrollment, graduation date, degree/certificate 
received, and major field of study for degree/certificate. While there was no consensus on this 
list, there was little disagreement with it either. 

4. Format for reporting graduation/completion rate data 

It was suggested that rather than having the institution calculate rales, institutions should provide 
both the numerators and denominators that would be neces.^ary for calcuialine 
gmduation/completion, transfer, or re-enrollment rates separately for each reporting category and 
^JJ 104)' ""'^^^ '"'^"''''^ ^"-S" ^'"^^"^ sender or race/ethniciTy under 
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"Safe Harbors" 



The group also addressed the question of what guidance could be provided to postsecondary 
institutions immediately to help them meet the intent of PL 101-542. Participams were asked 
to suggest specific actions institutions could take to be in compliance with PL 101-542 prior to 
the issuance of formal regulations by the Education Department (ED). This resulted in the 
following recommendations; 

1. Institutions should establish a cohort of first-time, full-time, degree-seeking 
students as of July 1, 1991 (fall, 1991). 

2. Schools with only short programs of 9-months or less in length should 
determine and have available for reporting in 1993 graduation/completion 
rates for the 1991-1992 academic year (July 1, 1991-Junc 30, 1992), 

3. Schools whose predominant programs are a year in length should determine 
and have available for reporting graduation/completion rates of the 1991 
cohort by July 1, 1993. 

4. Schools whose predominant programs are two years in length should 
determine and have available for reporting graduation/completion rates of the 
1991 cohort by July 1, 1994. 

5. Schools whose predominant programs are four years in length should 
determine and have available for reporting graduation/completion rates of the 
1991 cohort by July 1, 1997 or 1998. 

6. Schools that cannot report graduation/completion rates by the first reporting 
year specified in PL 101-542 (1993) should have available for reporting in 
1993 the percent of students in the 1991 cohort who are still enrolled in the 
spring of 1993 (the re-enrollment or persistence rate of the 1991 cohort). It 
will do the same for each succeeding cohort until graduation/completion rates 
can be fully reported. 

7. Schools that have the capability of reporting graduation/completion rates for 
their first-time, full-time degree-seeking students for an appropriate cohort 
in 1993 (a 1990 cohort for associate degree recipients and a 1986 or 1987 
cohort for baccalaureate degree recipients) should determine these rates and 
have them available for reporting by July 1, 1993. 
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PPENDIX B; TECHNICAT. NinT KR 

This survey was conducted through the Higher 
Education Panel, part of an ongoing survey research 
program created in 1971 by the American Council 
on Education. 

The panel is a disproportionate stratified sample 
of 1.040 colleges and universities, divided into two 
half-samples of 520 institutions each. The sample 
was drawn from the more than 3,200 institutions 
listed in the Education Directory, Colleges and 
Universities issued by the U.S. Department of 
Education. The Panel's stratification design (Table 
B-1) is based primarily upon institutional type, con- 
trol, and size. For any given survev . either the entire 
Panel, a half-sample or an appropriate subgroup is 
used. 

The sample for this survey consisted of 444 insti- 
tutions in one of the half-samples, excluding spe- 
cialized institutions (e.g.. rabbinical seminaries and 
schools of art) and institutions that offer no general 
program of undergraduate instruction. The same 
half-sample has been used in all Campus Trends 
surveys. 

The questionnaire {see Appendix A) was mailed 
un February 19. 1991 with the request that it be 
completed by the academic vice-president. By late 
May. responses were received from 81 percent of 
those surveyed. Actual respondents included; 
provosts, deans, or academic vice-presidents, 62 
percent: associate dean or associate provosts. 9 per- 
cent; presidents, 10 percent; and other. 18 percent. 

Data from responding mstitutions were statisti- 
cally weighted to be representative of the 2.379 non- 
specialized colleges and universities in the U.S. that 



Table B2~Response Rate 
by Institutional Categories 



Control and Type of Institution 



Response Rate 



Total 

Control 
Public 

Independent 

Type and Control 
Public Doctoral University 
Independent Doctoral University 
Public Comprehensive University 
independent Comprehensive University 
Public Baccalaureate College 
Independent Baccalaureate College 
Public Two-Year College 
Independent Two-Year College 

Enrollment Size (FTE) 
Less than 1 .000 
1.000 to 4.999 
5.000 to 9,999 
10.000 and above 



8f 

80 
82 

79 
7$ 
80 
83 
95 
83 
79 
85 

83 
80 
82 
80 



offer a general program of undergraduate instruc- 
tion. The weighting technique adju.sts the data for 
institutional nonresponse within each stratification 
cell. 

Table B-2 shows response rates by institutional 
categories. The lowest rate of response was among 
independent doctoral universities. 



Table B-1 Stratification Design 



Enrollment 
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1 

2 
3 
4 

7 
8 

9 

10 

11 

12 

13 

17 

18 

19 

20 

21 

22 

23 



Total 

Large putjlic doctorate-granting 

Large independent (toctoraie granting 

Large public comprehensive 

Large independent comprehensive 

Large pubifc two-year 

PuWic comprehensive 

Public comp-ehensive 

Indapenctent comprehensive 

Public baccalaureate 

Independent baccalaureate 

Indepenttont baccalaure^ 

PuWic 2-yeaf acsKfamic/comprehensive 

Public 2-year academic/comprehensive 

PuWic 2-year acMfemic/comprehenstve 

Pubftc 2-year Kattemic/comprehensive 

In^pendent 2-y8ar academic/comprehensive 

Public two-year occupational 

PuWtc two-year occupational 



a 
a 
a 
a 
a 

5,500-8,999 
<5.500 
<9,000 
<9,000 
1,350-8,999 
< 1,350 
6,000-8.999 
4.000-5.999 
2.000-3 999 
<2,000 
<:9.000 
2.500-8.999 
<2.500 



Population 


Sample 


Respondents 


2379 


444 


359 


104 


52 


41 


58 


29 


22 


92 


46 


39 


26 


12 


12 


43 


20 


16 


56 


18 


13 


108 


22 


17 


-12$ 


24 


18 


127 


21 


20 


166 


26 • 


21 


446 


3B 


32 


55 


18 


15 


72 


18 


14 


155 


24 


17 


332 


25 


20 


129 


13 


11 


63 


16 


14 


221 


22 


17 



n7f^7rX,^T^lZr'\°^ T "^•^^"^^ full-time equn.a/ent (FTE) 1981 enrollment greater than 8.999 (b) 

FTE 1981 graduate enrollment greater than 749: (c> FY 1979 educaftonal and general expenditures of $35 million or more. 
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TABLE 12 - Current Status of Student Assessment 



(Percentage of Institutions) 





Total 




Bacca- Compre- Doctoi«: 
laureate hensive 


rUDlIC 

2-year 


Public All 
4-vear Indeoendent 


Our institution curuBntlv has assessment 


















activities underway 


81 


88 


76 


82 


58 




85 


70 


Our institution has a separate budget (or 


















appropriation) for assessment 


28 


32 


21 


34 


22 


O A 
OH 


36 


18 


Assessment is part of a self-study for a 
















reoional accreditino aaencv 


88 
o o 


D D 


73 


72 


46 


65 


70 


70 


Assessment is oart of self-studies for 
















specialized accrediting agencies 


56 


*J o 


54 


66 


36 


C 7 

0 ( 


66 


*♦ J 


Our institution is developing: 
















* its own assessment instruments 


69 


70 


67 


79 


52 


67 


74 


70 


• methods of portfolio assessment 


45 


44 


46 


51 


36 


45 


44 


46 


Interest in assessment has decreased 


1 i 
1 i 


1 n 


12 


12 


12 


1 0 


14 


10 


Assessment has led to orooram or 
















curriculum changes 


52 


u o 


38 


43 


30 


by 


45 


39 


So far, assessment has mainly resulted in 
















new reporting requirements 


42 


■t £ 


42 


46 


29 


A O 

4 J 


45 


39 


Percentage of our (full-time) teaching 


















faculty who have oarticioated in assessment* 


















90 - 100 percent 


16 


15 


21 


10 


6 


1 5 


15 


1 7 


60 - 89 percent 


12 


1 1 


18 


6 


5 


10 


10 


15 


40 - 59 percent 


9 


13 


7 


3 


0 


1 1 


2 


1 1 


21 - 39 percent 


18 


20 


1 5 


20 


8 


22 


10 


1 7 


1 1 - 20 percent 


12 


1 1 


12 


18 


8 


9 


14 


14 


1-10 percent 


29 


26 


25 


43 


46 


28 


45 


23 


None 


4 


4 


3 


1 


27 


4 


4 


4 



Source Campus Trends. t99l, American Counctt on Education 



Weighted survey data ( 81 percent response) received from 3&9 institutions {inctudtng 124 two year coiteges. 73 tJaccalaureate institutions. 99 comprehensive universities, f ) ,1 
and 63 doctoral institutions^ % 'I 



TABLE 13 ~ Areas of Student Learning being Assessed * 



(Percentage of InstftuUons) 





Total 




Bacca- Compre- Doctoral 


Public 


Public 


All 








faureat© 


hensive 




2-year 4-year Independent 


Assessment Is tn plara for: 


















Basic coileoe-levei skills 


72 




65 


53 


41 


90 


64 


5B 


KnowledaG in OBneral education sublets 


o o 


O O 


35 


31 


25 


39 


38 


29 


Knowledge in a major 


36 


28 


44 


47 


23 


28 


4 1 


42 


Attainment of higher-order skills in: 


















critical thinkirta 


OA 


£. D 


24 


24 


1 6 




25 


23 


Quantitative oroblBni-solvina 


IP 




26 


34 


23 


36 


32 


27 


oral communication 


?ft 
^ o 




27 


27 


1 8 


28 


27 


27 


writinQ 


D O 


D D 


57 


55 


32 


C D 


49 


57 


Chdnfifi^ in ^tiiriAnt i/^thioc dnH attiturloc 

wiiaii^99 III oiuviviil Cll IW ClllflUUC7a 


O < 

C T 


0 


3 1 


29 


29 


0 


26 


32 


Lono-term outcomes of oraduates 




£ Q 


30 


33 


29 




3 f 


33 


Assessment is plann^l for: 


















Basic coiieoe-level skills 


1 7 


1 n 

1 u 


20 


29 


1 9 


8 


2 1 


23 


Knowledoe in oeneral educaticn subiect*; 


o *♦ 




28 


«? o 




35 


34 


34 


Knowledge in a major 


31 


31 


33 


27 


30 


30 


32 


32 


Attainment of higher-order skills in: 


















critical thinking 


43 


49 


40 


38 


27 


47 


41 


39 


quantitative problem-solving 


40 


46 


39 


36 


19 


44 


37 


38 


oral communication 


37 


40 


39 


32 


21 


39 


33 


38 


writing 


27 


30 


26 


24 


21 


28 


27 


25 


Changes in student values and attitudes 


35 


42 


29 


32 


26 


39 


33 


31 


Long-term outcomes of graduates 


45 


47 


48 


39 


29 


50 


42 


41 



• Responses tcr place." "planned," and "no' sum to 100 percent Percentages for "no change" are not shown on the tabte 



Source: Campos Trends. 1991 . Amer«:an Council on Education 

Weighted survey data ( 81 percent response) received from 359 mstilutions (including 124 two-year colleges. 73 baccalaureate institutions. 99 comprehensive universities, 
and 63 doctoral institutions) 
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TABLE 14 - State Mandates for Assessment Infonnation 



(Percentage of Institutions) 



Total 2-year 



For ail institutions: 

•Percentage wrth a state mandate to conduct assessment 
•Percentage required to submit assessment data to a state 
agency 

Experiences, among those public institutions required to submit to 
assessment data:* 
Our institution has submitted data 
This information has been made public 
This information is comparable across institutions 
The information to be reported includes. 

Description of assessment activity 

Summary, next steps planned 

Results on English or writing sitills 

Results on Math skills 

Results from standardized testing 

Department by department results 

Findings from alumni and/or employer surveys 

Submission of assessment Information affects our institution's 
funding:' 
No 

Ves, based on whether information Is submitted 

Yes, based on satisfactory results 



Bacca- Compre- Doctoral 
laureate hensive 



Public Public All 



1 c 

3d 


50 


1 4 


33 


31 


56 


51 


S 


25 


36 


9 


24 


23 


40 


36 


4 


78 


81 


60 


71 


73 


81 






74 


74 


40 


86 


77 


74 


74 




71 


78 


40 


59 


46 


78 


52 




90 


87 


100 


95 


100 


87 


97 




84 


79 


100 


90 


100 


79 


94 




65 


68 


60 


61 


46 


68 


57 




64 


68 


40 


61 


46 


68 


53 




61 


61 


50 


64 


50 


61 


59 




43 


36 


50 


58 


69 


36 


59 




62 


58 


75 


78 


64 


58 


74 




76 


77 


100 


64 


77 


77 


74 


* 


13 


14 


0 


18 


8 


14 


12 


• 


10 


9 


0 


18 


15 


9 


14 


* 



^^.^^ ,^ n^x^^nnnn ..wum^uns. i-iyrceniages snown nere are oaseo on puDUc insWutwns with a reauirement to submrt assessment data 

Soufce: Campus Trends. 199! , American Council on Edi«atwi. 

Werghted sunrey data ( 81 percent response) received from 359 institutions {'nduding 124 two-year coftefles. 73 baccalaureate institutions. 99 compretwisfve universities, 
and 63 doctoral Institutions) 



APPENDIX C 



Technical Notes and Additional Tables for the NCES 
Fast Response Survey of Institutional 
Capability for Reporting Student Outcomes 

SURVEY OF UNDERGRADUATE REPORTING CAPABILITY 
Survey Methodology and Data Reliability 

Questionnaires were mailed to Registrars at a national probability sample of postsecondarv 
institutions ,n early May of 1991. A modification of the 1990 Integrated Postsecondarv 
''^'oPrle' f^'''"" ^'^'y of postsecondary institutions, the mos't 

™^.c IT Estimates available ft^r use at the time, was tised for this survey The 

IPEDS Early Estimates siimple contains 1100 institutions and is divided into 9 sectors 
representing institutional control (public; private, nonprofit; private, for-profit) by level (4-year 
2-year, less-than-2-year). For example, sector 1 contains public, 4-year institutions and sector 
5 contains private, nonprofit, 2 year institutions. Because the 4-year and 2-year public 
institutions h.d been oversampled for the IPEDS survey and oversampling of those sectors was 
not necessary for the survey of reporting capability, every second institution in those two sectors 
sector 1 and sector 4, was selected for inclusion in the fast response survey sample, which then 
contained 866 institutions. 

During data collection, some institutions included in the .sjimple were found to be ineliuible 
Uraduate-only institutions, most of them private, nonprofit schools, were not eligible for this 
survey of undergraduate reporting capability. In addition, some institutions were ibund to have 
closed since the 1990 IPEDS Early Estimates sample was drawn. Ultimaielv, 788 of the sampled 
institutions were determined to be eligible. ' 

Data collection continued though May and June, 1991 with telephone folhvw-up of non- 
respondents beginning in early June. By the end of data collection on July 9, 1991 751 of the 
eligible institutions had responded, yielding an overall response rate of 95 percent The response 
rate for public institutions was 97 percent; for private, nonprofit institutions, 98 percent; and fi)r 
private, for-profit institution.s, 90 percent (table C-l). 

The survey data were weighted to produce national estimates. Weights were adjusted for unit 
non-response. The weighted number of in.stitutions in the universe is 9,93.\ Numbers in the 
tables and text have been rounded; however, percentages have been calculated on the actual 
estimates rather than the rounded vf.'ues. 
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Table C-1 Rc»pon$e rales of portsc^ndaiy lAsUtittiOfis to the Fast Rciponw Survey System on inttltuUonal capabUiiic 



by type of response and control and levd of in^itution: 1991 

Type of Response 



L-oniroi Dno ievc4 




Out Af 


Totfll in 


ConinlBlod 


Cpnpleted 




Nod 


Response 


of insiilution 


Total 


scope 


•cope 


by mail 


by pboiw 


Refused 


r^pondest 


rate 


Total 


866 


78 


788 


490 


261 


22 


15 


95.3 


Public 


294 


8 


2U 


201 


77 


2 


6 


97.2 


4-year-and-abovc 


92 


S 


87 


«9 


18 


0 


0 


100.0 


2"year 


149 


3 


146 


97 


43 


1 


5 


95.9 




53 


0 


53 


35 


16 


1 


1 


96.2 


Private, nonprofit 


323 


53 


270 


18} 


83 


2 


4 


97.8 


4 ^ycar -end-above 


257 


46 


211 


139 


68 


1 


3 


98.1 


2-ycar-and-bclow 


66 


7 


59 


42 


15 


1 


1 


96.6 


Private for-profit 


249 


17 


232 


108 


toi 


18 


5 


90.1 


2-ycar-and^abovc 


93 


3 


90 


45 


37 


5 


3 


91. t 


Lcss-tJ\ai^-2-ycar 


156 


14 


142 


63 


64 


13 


2 


894 



SOURCE: U S. Department of Education, National Center for Education Statistics* Fast Response Survey System. 
Poi^tsccondary Institution Reporting Capability, FRSS 43, 1991 
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The tindmgs in this report are subject to sampling variability. If a different sample of insiiiutions 
had been surveyed, the responses would not have been identical. The coefficient of variation 
(CV) is a measure of the variability due to sampling. It is the percent of the estimate that is due 
to sampling variability and indicates how much variance there is in the population of possible 
estimates of a parameter for a given sample size. CVs may be converted to standard errors by 
multiplying an estimate by its CV, converted to decimal fraction. Standard errors can be to 
develop a confidence interval for a given estimate. If all possible samples were surveyed under 
similar conditions, an interval of 1.96 standard errors below to 1.96 standard errors above the 
estimate would include the true population parameter being estimated in about 95 percent of the 
samples. This is a 95 percent confidence interval. For example, the estimate for the percent of 
all institutions able to report graduation-completion rates is 79.1 percent. The CV for this 
estimate 3.7 percent (table C-S). Dividing the CV by 100 and multiplying the resulting decimal 
fraction by the estimate (79.1 percent) results in a standard error of 2.9. The 95 percent 
confidence mterval for this statistic extends from 79.1 minus (2.9 times 1.96) to 79 1 plus (2 9 
times 1.96) or from 73.4 to 84.8 percent. This means that we can be confident that this interval 
contains the true population parameter 95 percent of the time. 

Estimates of CVs were computed using a balanced half sampling technique known as balanced 
repeated replications. Estimated standard errors for each of the variables included in Section III 
of this report are contained in tables C-2 through C-S. CV\s for statistics not included ihese 
tables may be obtained upon request. 

For categorical data, relationships between variables with 2 or more levels have been tested usinu 
chi-square tests at the .05 level of significance, adjusted for design effect. If the overall chi- 
square test was significant, it was followed up with pair-wise tests using a Bonferroni / statistic 
which maintained an overall 95 percent confidence level or better. 

Survey data are also subject to errors of reporting and errors made in the collection of the data 
rhese non-samplmg errors can sometimes bias the data. While general sampling theory can be 
used to deternime how lu estimate the sampling variability of a statistic, non-samplini. e'rrors are 
no easy to measure and usually require that an experiment be conducted as part the data 
collection procedures or that data external to the study be used for validation. 

Non-sampling errors may include such problems as differences in the respondent's interpretation 
of the meaning of the questions, differences related to the particular time the survey was 
conducted, or errors in data preparation. During the design of the survey and survey pretest, an 
effort was made to check for consistency of interpretation of questions and to eliminate 
ambiguous items. The questionnaire was pretested with respondents like those who completed 
the survey and the questionnaire and instructions were extensively reviewed. Manual and 
machine editing of the questionnaires was conducted to check the data for accuracy and 

tevT -."k 'im "^''^ n^'s^'^g or inconsistent items were recontacted by telephone; data were 
keyed with 100 percent verification. 
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Tab<# C-2 CV S for table 





Can fepcrt 




Control and tevet 


graduation/ 








Inittat 


of institutfon 


ccNTiplstfon 




Race/ 


Attendance 


mafOf field 


Tolat 


fato 


Gender 


elhnicily Age 


ttatus 


of study 



Total 



Student chamcter isttcs 
Final Schooler 
major fie'd academic 
of study division 



Athletic 
financial 
afd sfatut 



General 
financial 
aid ftgfui 



Retl<tofKy 
itatus 



2S 



3.7 



4.0 



58 



5.t 



27 



55 



36 



7.2 



6.5 



$2 



PubHc 


1 7 


43 


M 


3.1 


5.5 


55 


6.3 


32 


5.0 


15.7 


6,2 


4,2 


4-yaAr-and*abov0 


48 


6 J 


3,5 


36 


10,2 


118 


86 


46 


9.2 


15.4 


156 


8.7 


2-year 


1 3 


82 


33 


40 


75 


7,7 


14 3 


39 


74 


26,6 


99 


6.5 


Lew-than-2-^year 


0.0 


42 


69 


14.0 


85 


126 


11 6 


12.4 


21.0 


0.0 


9.3 


8.9 


Private nonprofit 


24 


49 


4.2 


60 


69 


4.1 


8.5 


30 


8.9 


22.3 


11.2 


9.3 


4-yaar-and-ahave 


39 


5.4 


4.3 


62 


89 


64 


11.5 


43 


10.5 


21.5 


9,3 


7.6 


2-yeAr-And*t>elow 


53 


10.2 


7.5 


132 


11.3 


4.9 


11.4 


67 


ii5.e 


1007 


238 


18,6 


Private for-profit 


45 


5.4 


68 


11,2 


9.3 


3.9 


62 


53 


13.1 


552 


9.7 


7.6 


2*year-and-above 


0.3 


65 


13.3 


21,5 


160 


65 


61 


37 


15.9 


56.9 


14.9 


166 


teB«-than^2-year 


55 


65 


78 


126 


104 


49 


98 


65 


163 


0,0 


100 


86 



SOURCE: U.S. Department of Education. National Center for Education Stattstkrs. Fast Response Survey System. Postsecondary tnstHutlofifti Repoi1ln9 Capability. FRSS 43. 1991 



Remedial 
course 



10.9 

60 
14.9 

63 
16.4 

17,7 
196 

32,0 

23.6 
31,7 
26.6 



Other 



390 

29.7 
54.1 
57.2 
38.6 

58.5 
S0.3 
133.9 

63.7 
0.0 
633 
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Table for table HI 




Control and level 
oflnstitution 



Total 



I^ermlne 
employment 
outcomes 



Total 


2.5 


2.6 


Public 




5.4 


4-yeaf-and-above 


4.B 


15.8 


2-year 


1.3 


5.9 


Less«than-2-year 


0.0 


12.2 


Private nonprofit 


2.4 


7.9 


4-year.and-above 


3.9 


8.7 


2-year-and-below 


5.3 


11.2 


Private for-profit 


4.5 


2.9 


2-year-ancl-above 


0.3 


6.7 


Less-ihan-2-year 


5.5 


3.4 



Sample 
survey of 
graduates 



7.4 

5.8 
9.0 

7.5 
26.4 

10.9 
9.5 

21.7 

10.8 
20.5 
13.2 



State 
record 
systems 



33.8 

19.6 
0.0 
24.8 
23.8 

86.2 
106.4 
24.6 

48.4 
106.5 
48.3 



Source: U.S. Department of Education. National Center for Education Statistics 

FRSS Ts'^TsTi^""'^^ Institutional Reporting Capability. 



Other 



21.4 

36.9 
77.6 
56.7 
51.7 

29.2 
36.0 
38.4 

37.0 
48.6 
42.0 
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Table C-4.— CVS for table 111-3 



Control and level 
of institution 


Total 


Determine 
necesary/ 
appn^riate 
ticer>SK 


Method of collection 


Sample 

tufveyof 

graduates 


State 
record 
system 


Other 


Total 


2.5 


6.4 


11.8 


10.0 


13.8 


Public 


1.2 


6.0 


8.8 


8.7 


23.2 


4-year-and-above 


4.8 


lO.U 


25.7 


21.5 


59.2 


2-year 


1.3 


6.2 


12.1 


12.6 


29.6 


less-than-2-year 


0.0 


6.7 


39.8 


15.4 


44.2 


Private nonprofit 


2.4 


12.2 


20.6 


18.4 


25.1 


4-year-and-above 


3.9 


15.7 


17.3 


27.4 


32.9 


2-year-and-below 


5.3 


18.9 


88.4 


23.2 


36.8 


Private lor-profit 


4.5 


9.1 


21.5 


14.5 


35.8 


2*year-and-above 


0.3 


18.6 


39.7 


25.8 


58.8 


Less-than-2-year 


55 


9.2 


23.7 


15.2 


41.7 



SOURCE: U.S. Department of Education, National Center for Education Statistics. 

Fast Response Survey System, Postsecondary Institutional Reporting Capability. 
FRSS43. 1991 
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Table C-S.->C\rS for table III-4 



Control and idvet 


Can r^on 


Tw>e of Insiftutien transferred to 


Total graduation/ 


ln*state ln*stdte 


of Institution 


completion 


public private Out«of*state 




rates 


institutions institutions institutions 



Total 
Public 

4-year-8nd-*above 
2-year 

Le$s*than<-2-year 

Private nonprofit 
4-year-and-above 
2-year*and-beiow 

Private for-profit 
2-year-and-above 
Less- than ■>2^year 



2.5 


3.7 


16.7 


16.8 


18.3 


1.2 


4.3 


14 -» 


21.3 


22.2 


4.8 


6.1 


39.3 


101.3 


101.3 


1.3 


8.2 


16.5 


27.5 


29.4 


0.0 


4.2 


33.1 


33.1 


33.1 


2.4 


4.9 


21.1 


20.2 


20.9 


3.9 


5.4 


30.8 


29.6 


31,7 


5.3 


10.2 


40.0 


40.0 


40.0 


4.5 


5.4 


33.1 


27.5 


30.0 


0.3 


6.5 


52.8 


52.8 


50.0 


5.5 


6.5 


35.2 


28.7 


31.4 



SOURCE: U.S. Department of Education, National Center for Education Statistics. 

Fast Response Survey System, Postsecondary Institutional Reporting Capability 
FRSS43.1991 w K I. 



Because the respondents to this survey were both institutions of higher education and other 
postsecondary schools, such as for-profit vocational or trade schw)ls, the questionnaire items were 
worded to encompass the range of potential reporting capability and methods of record keeping. 
Following the pretest, several items were altered in an effort to improve clarity, in particular of 
questions that were pertinent to one category of institution but not to others. Nevertheless, some 
ambiguity remained. For example, an item asked institutions whether baccalaureate^ degree 
students, associate degree students, certificate students, and/or transfers were defined as 
graduates-completers at^ their institutions. The certificate category was included to reflect the 
credential signifying completion of a program that is conferred by many of the other 
poslsecondar> institutions. However, some institutions of higher education checked certificate 
students because they granted some certificates (e.g., teaching certificates) even though they did 
not count recipients of certificates as graduates nor did they include them in graduation- 
completion rates unless they also earned a baccalaureate or associate degree. The 
misinterpretation was ascertained through consistency checks with another item on the 
questionnaire that asked how long certificate students were tracked to determine graduation- 
completion status. 

Extensive telephone follow-up to retrieve missing data and to resolve inconsistent item response 
was required for this survey. This suggests that key terms were subject to varying interpretaticm 
by the higher education institutions and other postsecondary institutions. For example, some 
institutions equated a student unit record system with a transcript system used for tracking an 
individual student's progress toward graduation-completion. Since a transcript .system does not 
allow for tracking students who have stopped out of the program, it does not fit the strict 
definition of a student unit record system. Other institutions had difficulty with the concept of 
"cohort" even though a definition was provided on the survey instrument. Finally, there was 
varying interpretation of the question that asked for the institution's criteria for including students 
in a cohort to track graduation-completion rates. Nevertheless, inconsistent responses were 
satisfactorily resolved through telephone follow-up. 

For the majority of the questionnaire items, the non-response rate was less than I percent. 
Exceptions were three items that asked for .specific numbers from small sub-populations. For 
example, the number of years graduates-completers are tracked to determine if appropriate 
licenses or certifications are obtained was asked only of tho.se institutions that determine licensure 
outcomes and that use tracking as the method to determine them, a total of 128 institutions; the 
non-response rate for this item was 2.5 percent. Also exceeding a 1 percent non-response rate 
were the item asking if all students registered for credit were included in student cohorts tracked 
for graduation-completion rate and the item asking institutions that did not have a student unit 
record .system and/or track cohorts whether they expect to have that capability within five years, 
both of which had a 1.2 percent non-response rate. 

Data are presented for all postsecondary institutions and by control (public; private, nonprofit; 
and private, for-profit) and level (4-year, 2-year, and less-than-2-year) of the institution. 

Additional data can be found in tables C-6 through C-9. 

C-8 
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Tawe C-6--Percentaoe ofpostsecondary institutions that can report flraduatlon/compietion rates an^j method of 
calculation used, by control and level of Institution; 50 States and District of Columbia laa? 



Public 

4-year-and-above 
2- year 

Less-than-2-year 

Private nonprofit 
4-year-and-above 
2-year-and-below 



Can report 
graduation/ 
completion 
rates 



Method of calculation 




Retro^ectlvely 
tracking 
completion 
class 



2.055 
577 

1.195 
284 



72.9 
82.8 
63.6 
91.9 



58.4 
76.5 
46.2 
60.5 



23.6 
14.3 
28.3 
26.6 



NumbBt of 
completers to 
number of 
first-time 
entries 



11.3 
6.6 

15.3 
8.6 



6.8 

2.7 
10.2 
4.3 




SOURCE: U.S. Oeparimeni of Education. National Center for Edcuaiion Statistics 

Fast Response Sun^ey System. Postsecondary Institutional Reporting Capability. FRSS 43. 1991 
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T«hl. C 7 - P«,c»nt.B. «>! Inrttulten. with . itudtnl unH rword .y-m tht tr«k «ud.nt cohort. .mJ U»l uw »•« crilori. for 

SO Stat99 and Oistrtct of Columbia, 1M1 



Cotttre) and level 
of institution 



invlHulions 

with a Track 
•tutff nt unit ihidant 



Total 7.432 

Public 

4-yoar'^nd-abova Sdl 

2-year 852 

LeM-l^an-2-yesf 199 

Pf Wate nonprofit 1 ,827 

4-yeaT-and-above 1 ,071 

2-year-and-b0low 75© 



ft4.3 

M.1 
839 
42.2 

aa.o 

62.0 
66.7 

&5.3 



Attendance criteria 



Rnt-^ma 

Flrat-ytar Full^me Part*4ifna 



emoUmant criteria 

cSnnett Dn^aeor hkm-cradit 
anumbar wrtlflcila Ra»leter«* Non ^legrae 
of crttfta aaaWng tor ertdlt aaaKing 



•9,2 


97.5 


62.0 


6.0 


96^ 


42.1 


10.2 


90.9 


98.8 


62.6 


6.6 


84.6 


53.0 


11.5 


1CH>.0 


100.0 


72.1 


0.7 


83.5 


70^ 


11.3 


97J 


97.1 


58.6 


15.1 


81.9 


42.4 


107 


58.9 


100.0 


35.4 


00 


100.0 


17.7 


15.6 


89.5 


n.1 


49.8 


10.5 


95.4 


563 


11.1 


80.9 


M.I 


».7 


127 


96.7 


59.2 


15.2 


87.0 


99.0 


32.9 


68 


932 


51.8 


4.1 


86.8 


96.3 


67.9 


1.7 


100.0 


31.5 


9.3 


84.2 


98,4 


64.4 


5.6 


1000 


51.1 


11.0 


87,3 




68.6 


1.0 


1000 


28.0 


9.0 



Noies'percentaoe. by tt«r«.aneo «h1 o nrollment criteria »r« based on >n«itu.ion. that track tfudeni coi>ons. 
FercentaQes do ruj. add up to 100 becauM re^ondenta were able to choose more than one criteria. 

SOURCE U S Department of Educalkm. National Center for Edcuation SlatisticB 

Fast Response Survey System. Postsecondary Institutional Reporting Capability. FRSS 43. 199i 
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Tabft C.ft~l>rc«tigiiof inmmion nrfth i mudm imJt racortf tyittm that track cohorts and attar astablfahad 
cohorts iisad to Ottermine graduatlon-complttlon rates, by control and level of Institution: 
SO States and Oisirlct o» Columbia. 1991 »«««iui.on. 



Control and level 
of institution 



Institutions 

vWtha 
student unit 
record system 



Track 

student 

cohorts 



Total 


7.432 


54.3 


Public 


1,232 


54.1 


4-year-and-above 


381 


83.9 


2-year 


652 


42.2 


Less-than-2-y8ar 


199 


36.0 


Private nonprofit 


1.827 


62.0 


4-yeaf-and-abov© 


1,071 


66.7 


2-year-and-beiow 


756 


55.3 


Private for-profit 


4.373 


51.1 


2-year-and-abov© 


748 


45.0 


Les5-than-2-year 


3.625 


52.4 


Notes: Percentages for aiterfng established cohorts 



Dropplf^o students from established cohorts 



Change from 
full-tfme to 
l art-tlm e 

2^8 

10.9 
10.0 

e.8 

23.4 

22.8 
13.0 
39.4 

26.3 
34.8 
24.8 



Change from 
degree or 
certificate 
seeking to 
non-dsgrs8 or 
non-cMiftoate 
seekint 

24.0 

21.5 
15.4 

30.2 
15.6 

39.9 
40.2 
39.4 

16.7 
36.0 
13.3 



^Opor 
transfer 
out 

51.4 

29.9 
21.1 
35.5 
46.8 

61.6 
51.5 
78.9 

52.6 
55.1 
52.2 



Adding 
transfer 
students to 
ostabiished 
cohorts 



57.1 

35.4 
17.1 
43.9 
64.4 

46.7 
34.9 
66.9 

68.8 
74.2 
67.9 



SOURCE: 



U.S. Department of Education. National Center for Edcuation Statistics 
Fast Response Survey System. Postsecondary Institutional Reporting Capability. FRSS 43. 1991 



m 



m 



Table C-9.-Pefcentaoo of institulions that can report oraduatlon/completion rales, by student oraduaHoo/comptetlon 
status, and by control and level of ins titution: 50 State and DIst rtct of Columbia^ 1991 



Control and level 
of institution 



Total 



Can report 
graduation/ 
completion 
rates 



Student praduation/completlon status 



Baccalaureate 


Associate 


degree 


degree 


recepients 


recipients 


22.4 


23.7 


31.1 


54.0 


97.6 


45.1 




78.0 


62.9 


23.4 


94.9 


30.1 


19.4 


14.4 


1.3 


13.5 


7.2 


71.5 




1.2 



Certificate 
recipients 



Transfers Other 



Total 9.933 79.1 

Public 2.055 72.9 

4-year-and-above 577 82.8 

2.year 1.195 63.6 

Less-than -2-year 284 91.9 

Private nonprofit 2,548 77.4 

4-year-and-at)0ve 1,542 73.6 

2-yeaf-and-below 1,006 83.2 

Private for-profit 5,329 82.3 

2-year-and-above 913 84.3 

Less-than-2-year 4,417 8i 9 



81.0 

70.7 
20.9 
92.0 
100.0 

54.0 
27.5 
89.9 

95.7 
81.2 
100.0 



16.6 


2.7 


20.8 


4.7 


6.5 


3.5 


28.3 


5.5 


25.0 


4.9 


11.7 


2.9 


8.3 


2.6 


16.4 


3.4 


17.4 


1.8 


20.2 


3.5 


168 


1.5 



rates. . 
Percentages do not add up to 100 because respondents were able to choose more than one criteria. 

SOURCE: U.S. Department of Education. National Center for Edcuation Statistics, 

Fast Response Survey System. Postsecondary Institutional Reporting Capability, FRSS 43, 1991 
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